DEV Community: DASWU

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

DASWU — Tue, 29 Sep 2026 15:21:31 +0000

Following the support for 100-billion-file environments introduced in JuiceFS Enterprise Edition 5.3, v5.4 further improves JuiceFS across several dimensions for large-scale deployments. At this scale, even small per-operation overheads can add up to significant resource consumption. Once metadata is distributed across multiple zones, the system also needs to coordinate across nodes while maintaining performance, consistency, and stability.

To address these challenges, JuiceFS Enterprise Edition 5.4 introduces the following new capabilities and improvements:

Support for up to 1 million simultaneously mounted clients
Fast cloning of directories containing massive files
More than 90% RDMA bandwidth utilization
Up to 100% higher random-read performance under high concurrency
On-demand data synchronization for mirror file systems
More flexible caching strategies for a wider range of workloads
Multiple operational improvements for large-scale data

Supporting 1 million clients

As AI agents move toward large-scale deployment, the JuiceFS team faced a new challenge: supporting connections and concurrent access from up to 1 million clients on top of the metadata workload of a 100-billion-file file system. The team spent months exploring and optimizing the architecture, and introduced this capability in JuiceFS 5.4.

JuiceFS uses a distributed architecture that partitions metadata across multiple metadata nodes, allowing metadata for hundreds of billions of files to be managed across multiple zones. To allow these nodes to handle a large number of client connections at the same time, v5.4 co-locates metadata nodes with network proxies, enabling clients to connect to any metadata node.

The node that receives a request handles requests for its local partition directly. Requests involving other partitions are forwarded internally to the corresponding nodes. Multiple client sessions share the same physical connections between nodes, reducing the number of connections and the associated maintenance overhead.

Connection reuse is only part of the challenge. Querying large numbers of client sessions, reporting session status, and cleaning up expired sessions can also introduce significant overhead.

In v5.4, indexes are used to locate client sessions instead of scanning unrelated sessions. Paginated queries and sampled reporting reduce the overhead of session status management. Expired sessions are cleaned up in batches to prevent cleanup work from becoming concentrated in a single pass.

With these optimizations, JuiceFS 5.4 successfully handled 1 million simultaneous client connections in stress tests. Previously, when clients connected separately to multiple metadata nodes, the total number of connections could reach tens of millions. After optimization, 10 internal proxy nodes handled 1 million client connections, while connection reuse reduced the number of physical backend connections from the proxies to the metadata leader to about 400. This significantly reduced connection-management pressure on the leader.

Fast cloning for large directories

When creating branches of training data or preparing test environments, users often need an independently modifiable copy of a directory. As directory sizes grow, cloning becomes more complex: in a multi-zone architecture, the metadata for a single directory tree may span multiple metadata zones, so cloning must coordinate subtree operations across nodes while preserving the complete directory hierarchy.

v5.4 introduces cross-zone directory cloning. It recursively processes subtrees while preserving the original cross-zone layout of the directory tree. Cloning only copies metadata. The underlying object data remains shared, so there is no need to copy it. Subsequent writes to the clone do not affect the source files.

In a deployment with 30 metadata zones, cloning 100 million files took 1 minute and 40 seconds, averaging 1 million files per second. More zones can provide greater parallelism, while actual performance also depends on metadata load and how the directory tree is distributed across zones.

Note that cross-zone directory cloning is not atomic. If new data continues to be written to the source directory during cloning, different subtrees in the cloned directory may reflect the source directory at different points in time. If strict consistency is required, pause writes to the source directory before starting the clone.

Performance improvements

More than 90% RDMA bandwidth utilization

Starting with JuiceFS 5.3, clients can use RDMA to communicate with distributed cache nodes, reducing CPU overhead during data transfers and improving transfer efficiency. v5.4 further improves bandwidth utilization and transfer stability.

In a test environment with a 400 Gbps RDMA NIC on the client and an 800 Gbps RDMA NIC on the distributed cache node, a single client achieved 45 GB/s of read throughput. A single distributed cache node delivered 90 GB/s of data throughput, with NIC utilization exceeding 90%.

Higher random-read IOPS under high concurrency

In multi-process training, HPC, and rendering scenarios, high-concurrency random reads can stress not only storage and network resources, but also client lock contention and request-processing overhead.

Over the past six months, the JuiceFS team has continuously optimized high-concurrency random-read performance by simplifying frequently executed code paths and reducing lock contention.

As a result, random-read IOPS has more than doubled overall compared with the previous implementation. With distributed caching, a single client reached up to 189,000 IOPS. With local caching, a single client reached up to 156,000 IOPS.

Test environment: A dual-socket AMD server with 128 cores and 256 threads, 1 TB of DDR4 memory, a 100 Gbps NIC, and a 3.84 TB NVMe SSD used as the cache disk. The system ran Linux 5.14.

Optimizing data access across regions

On-demand synchronization to reduce data replication

In multi-region, multi-provider, and multi-cluster compute environments, a JuiceFS mirror file system can automatically synchronize files and directories from the source. Metadata is continuously synchronized to keep the directory structure and object versions up to date. Previously, object data was synchronized in full in the background by default.

However, a workload in the mirror region may only need a subset of the models, datasets, or historical directories stored at the source. For example, if the source contains 10 PB of data but the mirror accesses only 1% of it, full synchronization would transfer and store a large amount of data that the workload never uses.

v5.4 supports on-demand synchronization of object data for mirror file systems. When enabled, metadata continues to synchronize, while object data is no longer synchronized automatically in the background. When a client reads data, JuiceFS first checks the object storage in the mirror region. If the data has not been synchronized yet, JuiceFS fetches it from the source and stores it in the mirror on demand. This reduces unnecessary data transfer and storage consumption.

If an application requires local read performance on the first access, you can warm up the required data in advance. If a complete copy of the data is required, you can continue to use background synchronization.

Accelerating metadata access with read-only nodes

For workloads that only need to accelerate metadata access, v5.4 introduces a new mode that does not require creating a mirror file system.

Users can deploy read-only metadata service nodes in other regions to accelerate metadata reads for local clients. Clients continue to mount the source file system, while the acceleration nodes can scale dynamically to match access requirements.

More flexible cache management

Improving hot-data cache hit rates

Data from one-time scans is often unlikely to be accessed again. Caching this data can evict hot data that is accessed repeatedly. By default, when a request misses the distributed cache, the cache node fetches the data from object storage and stores it in the cache. v5.4 introduces a new cache-aside mode. When enabled, a cache miss causes the application node to read directly from object storage instead of filling the cache with the requested data.

Preserving cache hit rates when adding multiple cache nodes

When multiple cache nodes are added at once, read requests may be routed to new nodes that do not yet have the required data, triggering additional reads from object storage.

v5.4 introduces the --commission parameter. After a new node is enabled, it retains the node mapping from before scaling. If the data is not available in its local cache, the new node fetches it from the original node instead. This reduces reads from the source during cache scaling.

Reducing the number of files on large cache disks

On large-capacity cache disks, storing each cache block as a separate file can cause the number of files on the cache disk to grow continuously.

The new merge-cache mode combines multiple cache blocks into larger files and stores their indexes separately. This significantly reduces the number of files that the cache disk needs to manage.

This feature is currently available as a public beta.

Warming up specific byte ranges

If a workload reads only part of a large file, warming up the entire file consumes additional cache space and transfers unnecessary data.

v5.4 allows you to specify one or more byte ranges to warm up. JuiceFS only warms up cache blocks that intersect those ranges, aligning cache preparation with the data the workload actually needs to read.

This capability enables more precise warm-up for data formats such as Lance and Parquet.

Simplifying operations for massive files

Restoring files by deletion time

When restoring files from the trash, users may know approximately when an accidental deletion occurred but have difficulty identifying the target files by name or keyword alone.

The restore command now supports the start-time and end-time parameters. In addition to the existing keyword filters, these parameters allow users to restrict the restore operation by file deletion time.

Reducing GC and `fsck` memory usage

When checking large file systems, GC and fsck need to process large numbers of records, and sorting can consume significant amounts of memory.

v5.4 adds external sorting modes to both commands, using disk storage during sorting to reduce memory requirements. The external sorting mode for GC is available only with dry-run, allowing users to inspect and evaluate the operation without actually deleting data.

Restricting access with token allowlists

v5.4 supports access control through token allowlists. An allowlist can specify multiple first-level subdirectories under the root directory.

Clients using the token can only access and modify content within those subdirectories, allowing different teams or workloads to be restricted to their designated data.

Summary: scale changes everything

JuiceFS 5.3 brought the platform into a new phase of scale. At the same time, the rapid development of AI continues to push the boundaries of data volume and system load.

As file counts, client counts, and request volumes all reach new levels, performance, stability, consistency, and operational complexity are no longer independent concerns. They interact with one another and become increasingly difficult to manage as the system scales.

v5.4 addresses these system-level engineering challenges across large-scale deployments, covering client connectivity, cross-zone directories, data access, mirror synchronization, and cache management. JuiceFS Cloud Service users can now try JuiceFS Enterprise Edition 5.4 online. Users running on-premises deployments can contact the JuiceFS team for upgrade support.

JuiceFS is used across a range of demanding workloads, including LLMs and multimodal models, autonomous driving, embodied AI, and quantitative investment. We’ll continue working with organizations at the forefront of these fields to improve JuiceFS and build infrastructure that can keep pace with their applications.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

DASWU — Thu, 24 Sep 2026 01:58:23 +0000

Most conversations about AI sandboxes start and end with boot speed and compute isolation: How fast does the container boot? Can the agent run untrusted code safely? But these discussions miss a fundamental truth: an autonomous agent turns a fresh machine into a rambling digital workspace. It clones code repos, installs heavy node_modules or Python virtual environments, downloads browser binaries, compiles artifacts, generates multi-gigabyte datasets, and writes massive trace logs.

"The files are not incidental. They are the working memory of the task."

– Aniket Maurya, Founder @ Celesto AI

The naive solution is to overprovision the root disk. However, determining the size of a boot disk before a task exposes its scope is an ineffective effort. A "quick bug fix" might require a full Next.js build chain, while a "data benchmark" might produce multi-gigabyte Parquet tables. Coupling system state to workspace state creates a bottleneck.

Instead, an emerging architectural pattern is radical in its simplicity: decouple the system disk from the durable workspace. Provide a POSIX-compliant mount that acts like an infinite, persistent drive, backed entirely by affordable object storage. But how do you build such a system without introducing complex services that defeat the purpose of ephemeral sandboxes? Recently, we co-hosted an Office Hours session with Celesto AI, a platform building secure sandboxes for autonomous AI agents, and the webinar was a delightful exploration of exactly this challenge.

Storage	Purpose
Root disk	Operating system, runtime, package manager internals, and system-level state
Workspace	Repositories, generated files, datasets, build artifacts, logs, and project state

Here is the creative engineering stack making this possible: JuiceFS as the file system layer, SQLite as the embedded metadata engine, and Litestream as the durability glue.

The creative trio: JuiceFS + SQLite + Litestream

Object stores (S3, GCS) are excellent for storing files and performing key-based lookups, but they are terrible at listing directories, handling stat calls, or managing millions of small files in batches (i.e., renaming a directory). You need a metadata engine to act as the "brain" of the file system, mapping paths to block locations.

Usually, JuiceFS deployments use centralized or clustered metadata services, such as Redis, TiKV, or FoundationDB. For isolated AI sandboxes, a creative alternative leverages the three familiar JuiceFS components working in harmony:

The JuiceFS client provides the POSIX bridge. It is responsible for the complex task of communicating with both the metadata engine and the object store. It is the "worker" that reads and writes the actual data, as well as performs administrative tasks such as status checks and data compaction.
SQLite acts as the metadata engine. Instead of connecting to a remote metadata server, JuiceFS also supports using a local SQLite database file as its metadata store. This means the entire file system metadata (every inode, directory entry, and block pointer) lives inside a single, standard .db file on the local disk.
Litestream provides the durability backing for that SQLite file. Litestream continuously takes SQLite snapshots and streams the SQLite write-ahead log (WAL) frames to the same object store.

Here is the magic: When the sandbox is running, JuiceFS uses the local SQLite file to resolve ls, cd, and read operations. The actual file blocks are already safe in the object store. (Although, the sync-interval configuration needs to be taken into consideration.) When the user stops the sandbox, the local SQLite file might be discarded, but Litestream has already replicated the metadata engine to the object store.

When the sandbox resumes on a fresh VM (or a different host entirely), the system simply downloads the latest SQLite snapshot from the object store and replays the WAL. Within seconds, the local SQLite can be fully reconstructed. JuiceFS mounts it, points back to the same object store blocks, and the agent is looking at the exact same workspace directory it left behind.

The JuiceFS file system doesn't care which computer it runs on. But it does remember its folder structure and file pointers with the portable SQLite snapshots and WAL frames. With this setup, we turned the working memory of the AI agent into files and blocks in the object store, making it durable and accessible instead of locking it to a specific boot disk or block device.

An unconventional but effective combo

JuiceFS supports interchangeable metadata engines. Choosing SQLite + Litestream subverts the typical playbook by prioritizing simplicity and per-agent isolation over global scale. Here is why this particular stack fits the AI agent use case so well:

Easy to set up: While it is not completely hands-off (you still manage the JuiceFS client, the local SQLite instance, and Litestream's replication configuration like Celesto AI does), this stack is drastically simpler to bootstrap than a distributed database cluster.
Per-agent isolation: Each sandbox gets its own dedicated SQLite metadata file. This eliminates noisy-neighbor problems. Metadata contention simply does not happen across agents, because each one has its own independent "brain" living right next to its mount.
Performance that fits the scope: A single agent workload typically produces tens of thousands to a few million files. SQLite handles this range comfortably with low latency and minimal resource footprint. While this approach obviously does not scale to the hundreds of billions of files that JuiceFS Enterprise Edition can manage globally, it falls squarely into the comfortable zone for individual agent workspaces.

Of course, this approach comes with important trade-offs and considerations:

No cross-agent sharing: The file system is inherently tied to a single SQLite instance. Agents cannot mount and share the same workspace concurrently. If your use case requires collaborative access or shared datasets across sandboxes, you would need a different strategy.
Resume latency depends on object store speed: When a sandbox resumes, the system must download the latest SQLite snapshot from the object store and replay the WAL frames. This adds noticeable startup time compared to a hot-mounted local disk. Tuning Litestream's sync-interval involves a direct trade-off between durability guarantees and resume performance.
Single-writer limitation: SQLite is fundamentally a single-writer database. Heavy parallel writes from multiple processes within the same sandbox can become a bottleneck. For autonomous agents following a largely sequential workflow, this limitation is rarely a practical issue. But it is worth keeping in mind for highly concurrent, multi-threaded agent designs.

The stack is not a universal solution. It is a deliberate, opinionated choice that trades global scalability and shared access for extreme simplicity and tight per-agent isolation. And for the AI sandbox use case, that trade-off often makes perfect sense.

The performance reality

This architecture is beautiful, but it isn't magic. You are trading local SSD latency for object storage durability and infinite scale. The benchmarks reveal a clear pattern:

Small files need attention: Writing 5,000 tiny 4 KiB files to an object-backed file system involves heavy metadata churn. Even with SQLite's fast local access, every close() and fsync() triggers WAL writes and Litestream replication. In these scenarios, the local root disk (ext4/XFS) will outperform the durable mount by a factor of 2x to 5x. See Celesto's blog post for more details.
Large sequential workloads shine: For datasets, model checkpoints, build artifacts, and logs (files > 10 MB), the object-backed mount achieves upwards of 200 MiB/s, which is nearly 90% of the performance of the local disk. The background block uploads are streamed efficiently. See Celesto's blog post for more details.

The golden rule for agents is to use the object-backed mount drive for durable, long-term state reports and large artifacts. Use the local ephemeral disk for high-churn, temporary scratch work that doesn't need to survive a restart. This hybrid approach gives you infinite capacity without sacrificing performance where it truly hurts.

Final thoughts

Cloudflare CFO Thomas Seifert recently predicted that within five years, non-human traffic could be as much as 1,000 times human traffic, making humans "a rounding error on the internet." He caveated that he's "called it wrong at every point along the way," but the direction is hard to dispute: AI and machine traffic already overtook human traffic in May 2026.

If that is true for network packets, it is likely true for storage as well.

Every autonomous agent session produces a workspace. Multiply that by millions of agents running continuously, resuming, retrying, and branching. Human developers generate storage in bursts tied to working hours. Agents generate it constantly, at machine speed.

This raises an uncomfortable question: if non-human consumers are about to dominate storage demand, are we designing for the right consumer? The combo of JuiceFS + SQLite + Litestream is one answer, but not the only one. Other approaches worth watching include object storage as the primary interface, shared metadata engine with concurrent access at agent scale (i.e., the JuiceFS Enterprise Edition), copy-on-write workspace snapshots, and many others.

Portable, shareable, scalable file storage for AI agents is still in its opening moves.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Keeping GPUs Fed: How Meta's AI Storage Architecture Aligns with JuiceFS

DASWU — Fri, 18 Sep 2026 07:18:00 +0000

Introduction

Recently, I came across an eye-opening blog post and presentation from Meta's engineering team detailing their AI storage blueprint at scale. Meta operates hundreds of exabyte-scale storage clusters serving everything from Facebook and Instagram to Meta AI.

What caught my attention wasn't just the scale—it was the architecture. After years of wrestling with AI's brutal latency and throughput demands, Meta has converged on a clear pattern: data-metadata separation, rich client design, and comprehensive caching.

Here's the fascinating part: this blueprint isn't unique to Meta. JuiceFS shares these same first principles of design. The hyperscaler's new construction closely resembles the cloud-native architecture that JuiceFS has advocated for years.

In this post, I'll analyze both architectures, compare their core components, dive into their cross-region capabilities, and explain why this convergence in architecture makes the most sense for a distributed file system.

Why storage matters for AI workloads

Before diving into the architectures, it's worth understanding why storage has become such a critical component for AI infrastructure.

Meta puts it succinctly: "If AI is the brain, storage is the memory: capability and speed are highly dependent on the size of memory and speed of retrieval." Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been far more modest. This disparity has made storage bottlenecks one of the primary contributors to GPU stalls for AI workloads, which is directly impacting both expenditures and time to market.

The math is brutal. During model training, hundreds of thousands of GPUs iterate over vast datasets across multiple epochs. GPUs synchronize their state periodically. If a single GPU stalls due to slow storage, it slows down the entire training job. In Meta's presentation, if GPUs were idle for about 20% of the time, "that 20% stall means about tens of millions of dollars loss every hour" for a data center the size of Manhattan.

Beyond the direct dollar cost, there is an even more consequential dimension: opportunity cost. The AI industry is moving at an unprecedented pace. Major model releases, which took roughly 104 weeks between 2020 and 2022, have now been compressed to approximately 4-week cycles by 2026. Every week of delay in shipping a frontier model translates to lost market share, diminished competitive advantage, and missed revenue opportunities.

In short, storage is what makes the difference between a model shipping in weeks versus months and between a billion-dollar data center running at 80% versus 100% efficiency.

Meta's AI storage blueprint: the evolved architecture

After iterating on their storage infrastructure, Meta has arrived at a definitive architectural blueprint to solve the GPU stall problem at its root while continuing to support existing external and internal products. The training stack has been gradually migrating to the BLOB-storage interface. The motivation is more about performance: AI workloads demand predictable, bounded pMax latency. Traditional layered metadata lookups could add hundreds of milliseconds per request, stalling GPUs during training. By moving to a BLOB-centric interface, Meta leverages flash drives to deliver the low, predictable latencies required to keep thousands of GPUs productive.

The three-component architecture

The diagram below illustrates the request flow for Meta's BLOB storage getObject API and shows the system's three major components.

The Tectonic data storage: At the foundation sits Tectonic, a regional, multi-tenant storage fabric providing high durability and availability through erasure coding. It supports tiering across HDDs and flash drives, intelligently managing hot, cold, and warm data placement.
The BLOB server and metadata layer: Operating on top of Tectonic, the BLOB layer handles API requests, stores metadata, and exposes a global, infinitely scalable storage fabric with policies allowing users to trade off between durability and availability, abstracting away the complexity of the underlying block storage.
The rich client SDK: Meta eliminated the data-plane proxy and built a rich client SDK with a Tectonic BlockClient embedded within it, enabling direct data reading from Tectonic storage servers to clients. The client also implements hedged reads and dynamic concurrency control to reduce tail latencies.

To maximize cache efficiency, Meta also leverages spare memory on GPU hosts as a distributed data cache by integrating peers from the Owl subsystem directly into the BLOB-storage client SDK, which yields an average cache hit rate of 80%.

Cross-region data access with tiered caching

Meta's cross-region approach treats storage as a global data lake, but with tiered caches accelerating access across geographical boundaries.

Cache Tier	Location	Role
L1 Cache	GPU host memory (RAM)	Hottest, most frequently accessed data
L2 Cache	GPU host flash (SSD)	Local flash tier for warm data
L3 Cache	Regional flash-based BLOB storage	Regional cache before falling back to global BLOB storage

The results speak for themselves: average ingestion time dropped by 93%, and worst-case ingestion time dropped by 97%. This is what it looks like when storage stops being the bottleneck and starts enabling AI research velocity.

JuiceFS: cloud-native, AI-native from day one

For readers already familiar with JuiceFS, the parallels to Meta's design are immediately apparent. The architecture Meta converged upon after years of iteration shares the same core principles that have guided JuiceFS from day one.

JuiceFS was architected from the ground up on the data-metadata separation principle, which also consists of three core components:

Object storage: JuiceFS uses object storage as the reliable, scalable backend for actual data storage. It supports all major public cloud offerings (AWS S3, Alibaba OSS, Google Cloud Storage, etc.) as well as self-hosted solutions like Ceph and MinIO. This layer ensures that AI datasets, which can be as large as petabytes or exabytes, are safe, accessible, and can grow almost without limit.
Metadata engine: The metadata engine stores all file system metadata, including file names, sizes, permissions, directory structures, and the mapping between files and their underlying data blocks. JuiceFS supports multiple engine options, including Redis, MySQL, PostgreSQL, TiKV, and many others. For extremely large-scale deployments, JuiceFS Enterprise Edition offers a proprietary Raft-based distributed metadata engine that provides high availability and strong consistency across multiple nodes.
JuiceFS client: The client handles all file I/O logic, including data slicing, merging, caching, and POSIX semantics enforcement. It communicates with both the metadata engine and object storage directly, coordinating reads and writes intelligently.

Cross-region strategy: mirrors and replication

The JuiceFS Enterprise Edition also provides a native approach to support cross-region and cross-cloud data access:

Mirror file systems: Deploy a full metadata cluster and optionally an object storage replica in a mirror region. Metadata syncs automatically from the source region with approximately 1-second latency under normal network conditions.
Async data replication: JuiceFS supports cross-region and cross-cloud asynchronous replication with a one-to-many topology. Data written in the primary region is replicated to multiple mirror regions in the background, enabling global datasets without manual copying.
Distributed cache: JuiceFS clients on mounting hosts can form a distributed cache group using a consistent hashing ring, sharing cached data blocks with one another. This is particularly effective for model training, where datasets are repeatedly accessed across GPU nodes.
Smart read strategy: Clients preferentially read from the local region. If the data isn't fully synced yet, they seamlessly fall back to the source region.

This design resonates with Meta's global data lake + tiered cache approach but offers a more flexible, cloud-native model that works across public clouds, private data centers, and hybrid environments.

Architectural similarities

On the surface, Meta mainly exposes a BLOB-style interface, while JuiceFS primarily provides a POSIX file system. Yet beneath these protocol differences, the architectural parallels are noteworthy. Both systems converged on the same three-component design with comprehensive caching and cross-region capabilities:

Dimension	Meta's BLOB-Storage	JuiceFS
Data storage layer	Tectonic: regional block storage fabric with erasure coding, tiering across HDDs and flash	Object storage: S3, OSS, GCS, Ceph, MinIO, and others
Metadata layer	Unified metadata scheme backed by ZippyDB with O(1) chunk lookups	Metadata engine: Redis, MySQL, PostgreSQL, TiKV, JuiceFS Enterprise Edition
Rich client	"Fat client SDK" with embedded Tectonic BlockClient, streaming directly from storage servers	JuiceFS client: handles all I/O logic, slicing, merging, caching; supports FUSE, POSIX, Hadoop SDK, CSI, S3 Gateway
Distributed cache	Tiered caching layers	Memory + local cache + distributed cache group
Cross-region	Tiered caching with global BLOB storage data lakes	Primary region + mirror file systems

A concrete example: explicit cache hydration

One interesting example is how both systems handle proactive cache hydration, which is the practice of loading data into the cache before it's actually needed.

Meta exposes a prefetch() API through the BLOB-storage SDK, while JuiceFS provides a warmup subcommand. Both serve the same purpose: allowing the client side to signal to the storage system exactly which data will be needed in the near future so it can be preloaded into cache before the GPU finishes its current work.

This is especially critical in distributed training, where thousands of GPUs may simultaneously read the same dataset. Without cache hydration, the initial requests would have to read from the underlying storage (Tectonic for BLOB and object store for JuiceFS) with higher latency. With prefetch() or juicefs warmup, data is staged in the distributed cache before the GPUs need it. In essence, both systems recognize the limitations of network-based storage and passive caching, and they both provide explicit prefetching as an important tool to compensate for what the storage layer alone cannot guarantee.

Conclusion

Meta's AI storage blueprint is a testament to the architectural patterns that emerge when a hyperscaler evolves its data infrastructure to hundreds of exabytes. After years of iteration, they arrived at a design that's similar to what JuiceFS has championed from day one: data-metadata separation, a rich client that handles I/O logic directly, tiered caching, and cross-region capabilities.

The same architectural pattern is also available to teams of any size. JuiceFS brought it into the big data era and now brings it into the AI age, with the flexibility of multi-cloud cross-region strategies, distributed caching, and a rich ecosystem of access protocols.

Hopefully, storage is no longer the bottleneck in your AI training stack. At least, it doesn't have to be.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Giving AI Agents a Real File System

DASWU — Fri, 21 Aug 2026 08:03:03 +0000

The new root user

"Everything is a file."

For decades, this single sentence defined the core Unix philosophy. Today, the primary operator interacting with those files is no longer a human at a terminal but an autonomous LLM agent spinning up environments, editing code, and executing scripts.

Because LLMs were pre-trained on codebases, wiki pages, documentation, directory trees, and terminal logs, POSIX file system operations serve as the zero-friction API for agentic reasoning. Giving an agent access to a familiar workspace lets it leverage standard developer tools immediately.

The following is an example output of an AI agent performing code reviews:

# Verified, the following command returns only the method definition.
# Your change doesn't have any actual invocation of this method.
grep -r "store.addDelayedStaging" . --include="*.go"

However, while a classic POSIX interface is necessary, modern AI agents demand far more from their storage than standard local file systems were built to deliver. To support a massive number of ephemeral, fast-moving agent sessions, the underlying storage layer must evolve as well.

Architectural paradox

This shift introduces a fundamental storage paradox for AI infrastructure engineers:

At the surface layer: The agent requires a workspace that strictly behaves like a classic, local POSIX file system so that standard developer tools (git, python, grep, compilers, etc.) function seamlessly without custom glue code.
Under the hood: Traditional instance-bound storage breaks down when managing hundreds or thousands of ephemeral, short-lived, or parallel agent sessions. The underlying backend must provide remote snapshotting, consistent multi-agent concurrent access, elasticity, and scalability while still being compatible with that familiar file interface.

To build scalable infrastructure for autonomous AI agents, we must preserve the classic Unix interface while completely re-engineering the storage engine beneath it.

Why AI agents love the file system

If APIs are rigid software contracts, POSIX file systems are the open playground. Giving an LLM agent direct access to a file system workspace unlocks operational advantages that structured tool calls simply cannot match.

Offloading the context window

Even the most advanced models with million-token context windows experience severe latency, high costs, and performance or attention degradation when a massive repository or gigabytes of build logs are included in an LLM prompt. The file system serves as secondary memory, allowing the agent to store raw state externally and selectively read, seek, or stream only the relevant fragments it needs at runtime.

The universal abstraction

A file system requires zero protocol overhead when an agent has shell execution access, unlike external services or database integrations that need custom JSON schemas or Model Context Protocol (MCP) tool wrappers. LLMs already possess a profound understanding of standard file operations and directory navigation out of the box. Exposing a POSIX file system gives the agent a universal workspace across any language, framework, or file format, without needing to define or maintain a single custom tool schema.

Unlocking the Unix ecosystem

By granting an agent a file system, it immediately inherits decades of battle-tested CLI tools: grep, awk, git, sed, compilers, and static analyzers. Rather than inventing "AI-native code search," an agent can simply execute grep or analyze git diff using tools already optimized for speed and scale.

Database-driven metadata meets elastic storage

When building agentic platforms, the naive approach is to rely on local block storage attached directly to a compute instance (e.g., standard ephemeral instance disks or container volumes). While simple initially, instance-bound disks rapidly fail under the demands of autonomous agents:

Ephemeral state loss: When an agent container scales down, crashes, or migrates to another host, all workspace state and context stored on instance-tied disks are lost unless manually synced.
Concurrent data access: Multi-agent workflows often require sub-agents (e.g., a planner, coder, and reviewer) to operate on the same repository simultaneously. Instance-attached disks cannot safely allow multi-node concurrent reads and writes without file lock corruption or race conditions.
Non-native data backup: Compute-instance disks lack built-in data backup or snapshotting, which makes rolling back the environment after an agent error slow and expensive.

To overcome these constraints, modern agent file architectures decouple the metadata engine (directory trees, permissions, file locks, and state) from the data storage layer (file payloads).

By backing the metadata layer with specialized database engines and offloading raw file blocks to elastic object or remote storage, file system operations essentially become database queries. The agent gets the exact POSIX semantics its tools expect, while the infrastructure gains consistency guarantees, automatic backups, and infinite horizontal scale.

The ecosystem landscape

Innovative projects across the storage industry demonstrate how database backends are powering this new era of AI agent file storage. Here, we are exploring a few notable approaches.

Lightweight embedded approach: AgentFS with SQLite

AgentFS, developed by Turso, embodies the lightweight embedded approach within the database-as-file-system paradigm. Designed specifically for agent sandboxing, it stores every piece of runtime state (file operations, key-value entries, and tool call histories) in a single SQLite database. The underlying engine is Turso, a full rewrite of SQLite in Rust that adds native asynchronous I/O and concurrent access, with optional cloud sync for portability across environments.

AgentFS exposes a POSIX-like virtual file system via an SDK or a FUSE mount. This unified storage foundation delivers copy-on-write isolation for safe agent execution, instant SQL-queryable audit trails for debugging and compliance, and single-file snapshots that make agent state trivial to share, version, and reproduce.

Client-server transactional approach: TigerFS with PostgreSQL

TigerFS, developed by Timescale, mounts a PostgreSQL database as a transactional file system over FUSE (Linux) or NFS (macOS). Every file maps to a row, directories map to tables, and file contents become columns. Every write is a full ACID transaction, enabling multiple agents and humans to read and write the same files concurrently with true ACID guarantees.

Multi-agent task coordination becomes as simple as moving a file (`mv task.md ./doing/`), with PostgreSQL transactions natively preventing race conditions without custom API orchestration. TigerFS offers two modes: file‑first for workspaces with atomic writes and reversible savepoints, and data‑first for exploring existing databases with Unix tools like `ls`, `cat`, and `grep`, with filters pushed down as SQL. Every change is versioned and fully reversible, and the file system itself serves as the API.

Decoupled distributed approach: JuiceFS with object storage

JuiceFS takes a fundamentally different architectural path: it decouples metadata and data storage entirely. File data is split into chunks and stored in S3-compatible object storage, while metadata lives in a separate, pluggable engine, supporting Redis, MySQL, PostgreSQL, TiKV, SQLite, JuiceFS Enterprise Edition (EE) metadata engine, and more. This separation enables JuiceFS to deliver strict POSIX compliance alongside shared, multi-node concurrent access, allowing hundreds of agent instances to safely work across a shared workspace. JuiceFS can be connected using FUSE, Kubernetes CSI, or SDKs (like Python, Hadoop, and S3 gateway), which makes it a flexible storage option for agents as well as other AI processes, including data preparation, model training, and deployment in different regions. Every committed change is immediately visible across all mounts with strong consistency, and the system scales elastically with object storage economics.

Together, projects like AgentFS, TigerFS, and JuiceFS represent distinct architectural explorations in the quest to provide agents with a persistent, shareable file system. Each tackles the problem from a different angle, yet all share a common recognition: agents naturally work with files. There is no clear consensus on which approach will become the dominant standard, and perhaps no single winner exists. What is certain is that the file system is re-emerging as a core primitive for AI agent orchestration, precisely because the POSIX interface aligns so naturally with how agents read, write, and organize data, and ultimately because agents complete tasks by acting on data and files.

Agents' full toolbox

While a database-backed file system provides a zero-friction home for code and context, a file system alone isn't enough for complex enterprise workflows. Real-world agents also need to orchestrate across multiple systems of record, for example:

The file system: As discussed above, the file system serves as the primary workspace for source code, local logs, build artifacts, and raw data manipulation.
Databases & APIs: The structured state layer for transaction logs, vector search embeddings, and relational data that models need to query or update.
The Model Context Protocol (MCP): An open standard protocol that standardizes how LLM applications connect to external data sources and tools through MCP servers. It does not replace existing APIs. Instead, it provides a common interface and transport layer on top of them so clients can discover and use capabilities without custom integrations for every application or model.

By combining elastic file workspaces with standardized protocols like MCP, developers can build agents that are autonomous, practical, and powerful.

Final thoughts

We aren't just building sandboxes anymore. Instead, we are designing a new type of operating system. In this model, the LLM acts as the CPU, the context window is the fast L1 cache, and database-backed file systems serve as persistent NVMe drives.

The industry is still actively debating what an agent's ultimate memory architecture should look like: whether it should rely on vector databases, structured state stores, or file-based paradigms. While the ideal abstraction may continue to evolve, the file system is already proving to be a core foundation in agent infrastructure. Let's honor the classic Unix philosophy and give agents seamless access to decades of battle-tested software tools through a familiar POSIX interface and the performance and scale that modern distributed backends deliver.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

GPFS vs. Alluxio vs. JuiceFS: Architecture and Use Cases Compared

DASWU — Fri, 14 Aug 2026 03:58:21 +0000

In AI workloads, scenarios such as training, inference, model distribution, agents, and data lakes each have distinct requirements for throughput, latency, concurrent access, POSIX compatibility, consistency, cost, and operational complexity.

In this article, we’ll first examine the typical I/O patterns of AI workloads, identifying the practical challenges these workloads pose to storage systems. Then, we’ll compare General Parallel File System (GPFS), Alluxio, and JuiceFS, analyzing their architectural differences, file system semantics, caching mechanisms, cost models, and applicability across different AI scenarios.

I/O patterns and storage challenges in AI workloads

Based on our interactions with enterprises across various AI domains, we can categorize common requirements as follows.

Autonomous driving: large‑scale data generation and training

Autonomous driving is one of the most data‑intensive AI storage scenarios. Large fleets of data‑collection vehicles continuously generate images, videos, sensor data, and trajectory logs, which undergo cleaning, labeling, and format conversion before entering model training pipelines. Common data formats include .mcap, .pack, TFRecord, and LMDB. These environments require highly stable and scalable storage due to the long data pipelines, diverse formats, and heavy training workloads.

LLM training: full‑pipeline data access

Foundation model scenarios cover multiple stages, including data cleaning, model training, checkpoint reads/writes, and inference services. Model weights, training data, and intermediate results are accessed repeatedly across different stages. The storage system must sustain long‑running jobs while maintaining stable data access during task failures, node outages, and training recovery.

Multimodal models: coexistence of small files, aggregated data, and model files

AIGC scenarios include text‑to‑image, image‑to‑image, text‑to‑video, image‑to‑video, and 3D generation. Training inputs may consist of many individual images or video clips, or they may be aggregated into formats like LMDB or Parquet to improve training efficiency. The training process also generates checkpoints and outputs model files such as safetensors. This makes the data landscape more complex than in single‑task training scenarios.

Compute platforms: multi‑cloud collaboration and model distribution

Compute platforms focus on distributing models and data across multiple environments. Users may pull models from external repositories or upload their own, and then run training or inference workloads across different clusters and cloud environments. The key challenge is reducing redundant copying while enabling consistent access to the same data across different compute environments.

Quantitative finance: balancing performance and cost

Quantitative finance typically involves smaller data volumes than autonomous driving or AIGC. However, as techniques such as Transformer‑based models, neural network training, time‑series modeling, and market graph structure analysis are increasingly adopted, storage cost is becoming a more explicit selection factor.

In recent discussions with quantitative finance customers, we’ve observed growing attention to storage costs. High‑performance parallel file systems like GPFS are inherently performance‑oriented, especially in large‑capacity, all‑flash configurations. In some on‑premises deployments, the storage investment for a large‑capacity all‑flash GPFS cluster can approach the cost of an entire 5090 GPU cluster.

AI agents: data sharing in short‑lived sandboxes

AI agents represent a rapidly emerging scenario. They often involve many short‑lived sandboxes—each executing a subtask with a lifetime of just seconds or even less.

While these tasks run briefly, the context, model files, tool files, and intermediate results must be shared among subtasks. If each sandbox independently mounts a file system, and the mount process itself takes several seconds, task scheduling efficiency suffers. A more practical approach is to pre‑mount the file system on the host and then expose it to sandboxes via bind mounts, PVCs, or similar mechanisms.

From a storage perspective, AI agents are less concerned with sheer capacity and more focused on data continuity, shared access, and mount efficiency within short‑lived tasks. As agent complexity grows, demands on file system semantics and data sharing capabilities will continue to rise.

The table below shows typical I/O patterns and storage requirements for AI workloads:

Scenario	Typical I/O pattern	Key storage challenges	Selection criteria
Autonomous driving	Large‑file throughput, mmap random reads, small‑file reads	Large data scale, long training pipelines, high random‑read pressure	Throughput, caching, metadata capabilities, capacity cost
LLM training	Large‑file reads/writes, mixed reads, checkpoint reads/writes	Full‑pipeline access, high stability requirements for long‑running tasks	Stable throughput, concurrent access, fault recovery
Multimodal models	Small‑file reads, aggregated large‑file reads, model file access	Coexistence of small and large files, significant multi‑task concurrency	Caching, metadata management, multi‑task concurrency
Compute platforms	Model distribution, cross‑cluster access, multi‑cloud collaboration	Data must be consistently accessible across environments	Unified namespace, multi‑cloud distribution, cache governance
Quantitative finance	Large‑file sequential reads, small‑file reads, training/backtesting access	Rising cost sensitivity, need to balance performance and cost	Capacity cost, scalability, long‑term operations
AI agents	Small I/O, multi‑client sharing, short‑lived access	Mount efficiency, data continuity, task isolation	File system semantics, shared access, mount methods

GPFS vs. JuiceFS

From PFS to GPFS: capabilities and limitations of parallel file systems

To understand GPFS, it helps to first understand parallel file systems (PFSs). Intuitively, a parallel file system separates metadata and data, enabling multiple clients to access underlying storage resources in parallel. In this architecture, metadata and data travel different paths. Clients do not need to funnel all I/O through a single node; they can concurrently access underlying disks or storage nodes. This allows hundreds of compute nodes to simultaneously read and write block devices or storage resources, breaking the network bottleneck of a single path and enabling horizontal scaling.

GPFS, Lustre, and BeeGFS are all typical examples of parallel file systems.

GPFS, later renamed IBM Storage Scale, is a mature parallel file system that has been widely used in high-performance computing environments for many years. It provides strong advantages in throughput, concurrent access, and consistency, making it suitable for workloads that require high performance and reliability.

However, as AI infrastructure continues to scale and organizations increasingly prioritize cost optimization, the high cost and operational complexity of GPFS can become important considerations during storage selection.

Typical GPFS workloads include quantitative finance, genomic analysis, physics simulations, and weather forecasting.

Architecture advantages and trade-offs: metanode, token locks, and strong consistency

The architectural advantages of GPFS mainly come from two mechanisms:

Metanode, which affects metadata coordination
Distributed token locking, which determines how consistency is maintained during concurrent access

Metanode-based metadata coordination

Under the Metanode model, a GPFS cluster usually consists of:

I/O servers
data disks
metadata disks

Metadata is stored on dedicated metadata disks. Unlike systems with centralized metadata services, GPFS allows clients to participate in parts of metadata coordination.

In other words, GPFS assigns different clients as coordination nodes for different files or inodes instead of routing all metadata requests through a fixed centralized service.

Therefore, GPFS clients need to maintain communication with each other, typically through port 1191.

When network issues or connection failures occur, the cluster manager must determine node status and remove failed nodes from the cluster to prevent split-brain conditions and data inconsistency.

This design helps distribute metadata coordination pressure. Under stable network and storage conditions, it can support high levels of concurrent access.

However, it also introduces higher environmental requirements. If network quality is unstable or storage latency increases, coordination efficiency may degrade. In severe cases, users may experience:

Request stalls
Long waiters
Recovery operations requiring node restarts

These problems do not necessarily indicate a weakness of GPFS itself. Instead, they reflect that high-performance, strongly consistent architectures require reliable networking, disk, and cluster state management.

Distributed token locks

Distributed token locking is another key mechanism behind GPFS consistency.

GPFS manages read and write operations through tokens:

Read operations require read tokens
Write operations require write tokens

When multiple clients access the same file, GPFS controls concurrent access by granting, revoking, and transferring tokens.

For example, if Client A holds a write token for a file and Client B wants to read or modify the same file, GPFS must revoke the token from A. A must flush dirty data to disk and release the token before B can proceed. This ensures strong consistency, but it also relies on stable, fast network and disk responses.

If disk writes become slow or network communication is interrupted during token revocation, long waiters can appear. In practice, requests waiting on Revoke Token or Reopen Token operations are not uncommon. This illustrates GPFS' inherent trade‑off: token‑based strong consistency and concurrency control come at the cost of increased system complexity.

Early InfiniBand networks or high‑quality fiber networks could reliably support this mechanism due to their low latency and high reliability. However, in some newer deployments—particularly those using RoCE—suboptimal network conditions, hardware quality, or operational expertise can amplify stability challenges in token coordination, especially when hundreds of clients coexist in a single cluster.

Overall, this mechanism enables high‑performance parallel access but demands high network, disk stability, and cluster operational capabilities. Client anomalies also require handling token recovery and cluster state restoration. Therefore, it’s essential to assess whether your team has the necessary deployment, monitoring, and troubleshooting skills.

Based on practical experience, GPFS deployments also require attention to several engineering considerations:

mmap scenarios: GPFS employs special mechanisms for mmap, such as pagepool, to reduce memory copying and improve performance. However, this also creates complex interactions with OS memory management. Large‑scale reliance on mmap patterns is generally not recommended without thorough validation.
Hot files and large directories: Hot files, hot directories, massive small files, or many clients concurrently accessing the same directory can become performance bottlenecks due to metadata coordination and lock contention. Mitigation typically involves directory splitting, data sharding, and access pattern optimization.
Operational management and monitoring: GPFS' management interface is not particularly user‑friendly for newcomers, and some monitoring information is not intuitive. Many teams supplement with external monitoring systems like Grafana to better observe cluster state, performance metrics, and anomalies.
Cluster Export Services (CES), Active File Management (AFM), and other components: While these features enable more complex scenarios, they also introduce additional configuration, operational, and troubleshooting overhead. Teams without long‑term GPFS experience should carefully evaluate this complexity upfront.
Capacity planning: Many GPFS or CPFS deployments emphasize expansion over contraction. Capacity planning early on is crucial to avoid either excessive initial capacity (and long‑term cost) or insufficient capacity (hampering business growth).

Performance comparison

Intuitively, many people assume that JuiceFS—built on object storage and independent metadata services—cannot be directly compared to GPFS. However, in certain AI workloads, both systems do present comparable alternatives.

It’s important to note that GPFS performance depends heavily on the synergy of underlying disks, storage servers, networking, clients, and the parallel file system itself. JuiceFS, on the other hand, is influenced by object storage performance, metadata services, client caching, distributed cache groups, and mount modes. Therefore, comparisons should not be reduced to single performance numbers—they must account for I/O patterns, deployment architecture, data scale, and access paths.

The following tests were performed with JuiceFS Enterprise Edition. The Community Edition shares the same core architecture, so Community Edition users can also refer to these testing methodologies and results.

Sequential reads: GPFS better per node, JuiceFS scales through cache groups

In single‑node scenarios, the performance models differ markedly. In our tests, with two 400 Gbps NICs, a single GPFS node achieved about 100 GB/s for sequential reads.

Under TCP mode (200 Gbps NIC), a single JuiceFS node peaked at about 20 GB/s. With RDMA (two 400 Gbps NICs), it reached about 55 GB/s. For higher aggregate throughput, JuiceFS can scale horizontally by adding cache nodes. For example, in one autonomous driving customer deployment with about 150 cache nodes (each with 160 Gbps NICs), the aggregated application throughput reached about 2.3 TB/s.

Sequential writes: GPFS excels at synchronous writes; JuiceFS uses writeback for scale

For sequential writes, GPFS holds an advantage under synchronous write semantics. Its write capability stems from the underlying parallel storage system—data written is immediately accessible by other clients under strong consistency semantics. This is well suited for scenarios requiring write reliability, real‑time visibility, and consistency.

JuiceFS sequential write performance depends on whether writeback is enabled. In synchronous mode, writes go to the backend object store, and performance is constrained by backend storage, protocol overhead, and network latency. With writeback enabled, data is first written to client‑local cache and later uploaded asynchronously—aggregate throughput improves, but real‑time visibility and consistency semantics change.

Thus, GPFS is better for scenarios requiring synchronous persistence and real‑time visibility. JuiceFS with writeback is suitable for workloads that can tolerate asynchronous upload semantics.

Random reads: GPFS better at high concurrency, JuiceFS competitive in certain scenarios

In 4K single‑process random read tests, we compared JuiceFS, GPFS, and local disk /tmp (EXT4).

The figure below shows the test results:

At iodepth 1, 2, and 4, JuiceFS outperformed GPFS.
At iodepth 8, GPFS surpassed JuiceFS and stabilized around 80K IOPS.
JuiceFS peaked around 68K IOPS at iodepth 4 and 8 and then gradually declined at deeper I/O depths.

Multi‑process random reads present more complex behaviors. When multiple processes concurrently read the same file, GPFS' consistency and lock mechanisms can introduce extra overhead. To validate this, we tested both reading the same file and different files.

The following figure shows the test results:

GPFS reads of different files scaled rapidly with numjobs, reaching ~433K IOPS at numjobs=12.
GPFS reads of the same file dropped after numjobs=2, suggesting consistency coordination overhead.
JuiceFS reached ~258K IOPS at numjobs=12 and stabilized around 250K thereafter.

All tests were run with local caching disabled, and data was served from distributed cache. The results show that GPFS has higher random read performance under high concurrency, while JuiceFS remains highly capable—sufficient for most AI training requirements.

Random writes: GPFS superior at high concurrency

Random writes better expose architectural differences between the two systems. JuiceFS' random write performance depends on whether writeback is enabled. Without it, performance is mainly constrained by the object storage backend. With writeback enabled, performance reflects the client's local cache path capabilities.

Under writeback enabled and cache‑dir set to local NVMe storage, 4K multi‑process random write results showed:

From numjobs 1 to 3, JuiceFS increased from ~28K to ~56K IOPS, significantly higher than GPFS.
At numjobs=12, both were close (~51K vs. ~50K).
Beyond numjobs=16, GPFS continued to rise to ~66K (at 16) and ~81K (at 20), while JuiceFS was in the 50–60K range.

These results indicate GPFS holds an advantage in random write performance, while JuiceFS closes the gap significantly when writeback is enabled. However, random writes are not common in AI workloads, so this should not be a primary evaluation criterion.

Selection summary

The main differences between GPFS and JuiceFS are summarized in the table below:

Aspect	GPFS (symmetric, decentralized)	JuiceFS (separated metadata & data)
Metadata architecture	Distributed, embedded in local memory	Independent, high-performance external database (cloud-native, lightweight)
Data storage layer	Expensive, tightly coupled shared SAN / parallel disk	Cost-effective, highly reliable, highly elastic object storage
Locking & concurrency	Distributed token-based locks (strong consistency)	Optimistic concurrency (Community Edition) / Single-threaded core (Enterprise Edition)
Typical use cases	Select HPC and scientific computing scenarios	AI research, training, inference acceleration, large-scale data management

GPFS is better suited for environments with ample budgets and stringent requirements for low latency, strong consistency, high‑concurrency access, and random write capability—typical of traditional HPC, scientific computing, and some quantitative finance applications. Its performance also depends on stable networking, storage hardware, and professional operational expertise.

For AI research, training, inference acceleration, and large‑scale data management challenges involving scalability, performance, cost, and multi‑cloud management, JuiceFS is the more suitable solution—and has been validated in production across all the aforementioned domains.

Alluxio vs. JuiceFS

Alluxio is another solution that enterprises often compare with JuiceFS when evaluating AI storage architectures.

Both solutions can provide filesystem access and cache acceleration on top of object storage. However, they differ significantly in product positioning, data organization models, and metadata architectures.

From the perspective of product evolution, the two projects have taken different approaches:

Alluxio’s recent AI-focused development has mainly concentrated on its Enterprise AI product line. The latest release of the Alluxio open-source repository is v2.9.4, released in June 2024.
JuiceFS continues to evolve both its open-source and enterprise editions. Many new capabilities are first introduced, validated, and refined in the open-source edition before gradually becoming available in the enterprise edition to serve more production users.

Key architectural differences

Data organization and consistency boundaries

Alluxio uses a 1:1 transparent caching model, which preserves the original organization of files in the underlying storage system. Existing data does not need to be imported or reorganized in advance, and the caching layer can be added or removed as needed.

JuiceFS, on the other hand, splits files into data chunks and stores them in object storage. It uses a metadata service to maintain file system semantics and data chunk mappings. From the application perspective, users see a complete file system, while the objects stored in the backend object storage are JuiceFS-managed data chunks rather than the original files.

This difference also affects the source of truth and consistency boundaries:

For Alluxio, the actual data typically remains in object storage or other UFS systems. The cache layer mainly accelerates access. If applications bypass Alluxio and directly modify data in the underlying storage, additional cache invalidation, refresh, or synchronization mechanisms may be required.
JuiceFS combines metadata services and object storage into a complete file system. File states and data mappings are managed centrally by JuiceFS, and application reads and writes go through JuiceFS. Therefore, its consistency boundary exists within the file system itself and does not rely on additional synchronization between the cache layer and backend storage.

Cache and namespace organization

Alluxio emphasizes a unified namespace and shared cache pool. It can integrate multiple underlying storage systems, including OSS, S3, HDFS, Ceph, MinIO, and NAS, into a single namespace and accelerate access through a unified distributed cache.

For organizations that already have multiple storage systems and do not want to migrate or reorganize existing data, this approach provides greater flexibility. However, when multiple workloads share the same cache pool, resource management and isolation typically require policies such as directory-based controls, priorities, and time to live (TTL) settings.

JuiceFS typically uses an independent filesystem as the management unit. Different object storage backends, such as OSS, COS, and TOS, usually correspond to different file systems with their own cache configurations.

Compared with Alluxio’s emphasis on a unified view across multiple data sources and shared caching, JuiceFS focuses more on clear boundaries between file system management, permissions, and data governance.

In addition, JuiceFS Enterprise Edition can connect multiple buckets into a single file system and use the same cache resources for acceleration. This enables multi-source data access scenarios while maintaining clear file system-level management boundaries.

Metadata architecture

Alluxio Enterprise distributes caching and part of the state management responsibilities across workers. Workers not only provide data caching but also participate in managing cache status and data location information. This brings data access closer to compute nodes or GPU nodes. Coordination and management components still exist, but cache-related states are not entirely centralized in an independent metadata service. As a result, when workers frequently restart or scale in and out, additional attention is required for recovering cache states and coordinating data locations.

JuiceFS manages core file system metadata through an independent metadata service.

The open-source edition supports metadata engines such as Redis and TiKV, while the enterprise edition provides a highly available metadata service to support consistency and transactional operations. Distributed cache nodes are responsible only for caching data blocks. Therefore, cache node failures mainly affect cache hit rates and access performance, but they do not change file system metadata states.

How architectural differences affect real-world usage

POSIX compatibility

Alluxio exposes data from object storage or other UFS systems through file system interfaces, but its file system semantics are not always complete.

Capabilities such as timestamps, file locking, hard links, symbolic links, extended attributes, and ACLs may require additional configuration or have usage limitations. For read acceleration and lightweight access scenarios, these limitations may not be significant.

However, if an application intends to use Alluxio as a full-featured file system, these factors should be carefully evaluated.

JuiceFS aims to provide complete file system capabilities on top of object storage. Therefore, JuiceFS continuously improves POSIX compatibility, covering common file system semantics such as directories, permissions, timestamps, and file locks, as well as more advanced system calls such as ioctl-based immutable and read-only settings. In production environments, JuiceFS POSIX compatibility already covers the vast majority of workloads and approaches full POSIX compatibility.

Write performance and write amplification

Alluxio preserves the original file format, which provides transparent access and reduces the cost of onboarding existing datasets. However, random writes, overwrites, and partial updates require careful consideration of write amplification. The reason is that object storage generally does not support in-place modification of object contents. If the backend maintains a 1:1 file layout, a small modification may trigger rewriting a larger amount of data, potentially requiring the entire object or file to be rewritten.

JuiceFS uses a chunk-based data layout. For random writes or append operations, JuiceFS only needs to process affected data chunks and update corresponding metadata, without being constrained by the original object storage file layout. Therefore, JuiceFS can better control write amplification for partial updates and random write workloads. However, if users need to restore JuiceFS data into the original file format in object storage, additional export or migration operations are required.

Deployment and engineering capabilities

Alluxio Enterprise supports full-component Kubernetes deployment. It also supports temporary write caching without persisting data to object storage, such as:

temporary decompression workloads
intermediate computation results
short-lived data that can be discarded after use

These capabilities are closer to cache-layer acceleration and compute-side optimization scenarios.

JuiceFS Enterprise Edition currently typically deploys metadata services on virtual machines or physical servers, mainly due to requirements for metadata stability and system reliability.

JuiceFS focuses more on long-term filesystem capabilities, including:

large-scale metadata management
Trash and data recovery
transactional operations
cache governance
seamless upgrades
open-source ecosystem integration

In production, JuiceFS has already supported up to 500 billion files—an important demonstration of its capability as a complete file system in large‑scale data management scenarios.

Selection recommendations

The difference between Alluxio and JuiceFS is not simply about the number of features. They are designed to solve different problems:

If data already exists in object storage, HDFS, Ceph, MinIO, or NAS, and organizations do not want to migrate or reorganize existing data but only need a pluggable cache layer to accelerate reads, Alluxio can be a suitable choice.
JuiceFS is likely a better fit, if organizations want to build a complete file system on top of object storage while supporting:
- mixed read/write workloads
- full POSIX semantics
- strong consistency
- elastic metadata scaling
- tens of thousands of concurrent clients
- multi-cloud data management

Summary

GPFS, Alluxio, and JuiceFS address different core storage challenges:

GPFS focuses on low latency and high-concurrency read/write performance for high-performance computing workloads.
Alluxio primarily provides cache acceleration for existing datasets.
JuiceFS provides a complete, scalable file system built on top of object storage.

Therefore, storage selection should not be based only on individual performance metrics. Organizations should first determine whether they need high-performance shared storage, a data acceleration layer, or a complete file system designed for multi-cloud and large-scale data management.

The final decision should consider consistency requirements, data scale, cost, and operational complexity.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and the community on Discord.

5.5 Faster Small-File Writes: Tuhu Built a Unified AI Storage Platform with JuiceFS + Ceph RADOS

DASWU — Tue, 04 Aug 2026 02:46:16 +0000

TL;DR:

Tuhu leverages JuiceFS, TiKV, and Ceph RADOS to unify storage for AI training, AI inference, and analytics, supporting over 100 million files while achieving up to **5.5× faster small-file writes* and 3.2× faster small-file reads.*

Tuhu (9690.HK) is an integrated online and offline automotive service platform. By the end of 2025, we had grown to 162.3 million registered users and operated 8,008 service centers across China. As our business continued to expand, our infrastructure had to support increasingly diverse data management and compute workloads. Over the years, multiple storage systems—including NFS, Alluxio, MinIO, and SeaweedFS—had been deployed independently to meet different application requirements. While each solution addressed a specific use case, the overall storage architecture became fragmented, making data movement expensive and operations increasingly difficult.

When evaluating a unified storage architecture, we built a unified AI storage platform based on JuiceFS + TiKV + Ceph object storage cluster (RADOS), considering our existing private cloud infrastructure and Ceph deployment.

Today, this platform serves AI training, AI inference, and big data processing workloads through a single storage infrastructure. It currently manages over 100 million files. In low-concurrency benchmark tests, it achieved up to a 5.5x improvement in small-file sequential write performance and up to a 3.2x improvement in small-file read performance.

In this article, we’ll explain how we designed and deployed our unified AI storage platform on top of its existing Ceph infrastructure. We’ll also share several production optimization practices, including Ceph RADOS data paths, erasure-coded pools, write amplification reduction for small files, and stability improvements for containerized deployments.

From multiple storage systems to a unified storage foundation

In the early stages of our company's growth, storage infrastructure was built independently for different applications to maximize development speed and satisfy immediate application requirements. As the platform evolved, this approach gradually resulted in multiple storage systems coexisting in production.

The rapid adoption of cloud-native computing, large-scale AI training, and AI inference exposed the limitations of this architecture. Problems related to architectural consistency, data mobility, operational complexity, and storage performance became increasingly apparent.

We identified four major challenges.

High operational complexity: Production environments relied on numerous storage solutions, including NFS, Alluxio, MinIO, Ceph, SeaweedFS, and various cloud block storage services. Each system had its own deployment model, access protocol, operational workflow, and troubleshooting methodology. As storage clusters continued to grow, day-to-day maintenance, capacity planning, and version upgrades became increasingly difficult.
Data silos: Different storage systems exposed different interfaces and lacked a unified data access layer. AI training pipelines typically require POSIX-compliant file semantics, while surrounding tools for data processing, model management, backup, and archiving often depend on object storage APIs. Because these storage systems could not naturally share the same data, datasets frequently had to be copied or synchronized between platforms. This increased operational overhead, storage consumption, and workflow complexity.
Rising storage costs: As business continued to grow, both the number of files and total storage capacity increased rapidly. Meanwhile, infrastructure costs—including hardware procurement, datacenter resources, and storage operations—continued to rise. Improving storage utilization while maintaining performance and reliability became a key objective for the infrastructure team.
Higher performance requirements from AI workloads: AI training, AI inference, disaggregated storage and compute, and big data analytics placed higher demands on the underlying storage system's concurrency, throughput, access latency, and stability. In particular, large-scale small-file access, multi-node concurrent reads, model file loading, and online inference services required the storage system not only to provide stable capacity support but also to deliver predictable performance under high-concurrency conditions.

As a result, the existing architecture based on multiple independent storage systems could no longer support long-term application growth. We decided to consolidate fragmented storage resources into a unified storage foundation that would simplify operations while enabling seamless data sharing across different compute environments.

The new storage platform was designed around two major capability areas:

Block storage for infrastructure and virtualization workloads: This capability provided reliable, high-performance storage for virtual machine images, cloud disks, and traditional infrastructure services.
Unified data access for cloud-native workloads, including Kubernetes applications, traditional microservices, big data analytics, AI training, AI inference, and emerging middleware such as vector databases. Among these, AI training and inference posed more concentrated challenges in terms of file semantics, object interfaces, data throughput, small-file performance, and multi-node concurrent access.

Therefore, building a unified cloud storage foundation involved both block storage capability for virtualized environments and data access capability for AI and cloud-native workloads. The following section will focus on the latter—how to build a unified data access foundation based on JuiceFS to support AI training, AI inference, and big data processing.

Building a unified AI storage platform with JuiceFS

When designing a unified storage platform, we evaluated several architectural approaches based on our existing private cloud infrastructure, storage investments, and workload characteristics.

Our primary goals were to support large-scale data access, enable disaggregated storage and compute, provide multiple access protocols, and deliver high reliability without introducing unnecessary operational complexity.

Rather than building a new distributed storage system from scratch, we chose a modular architecture based on JuiceFS, TiKV, and Ceph RADOS. This approach allowed us to leverage our existing Ceph infrastructure while introducing a unified file system capable of serving AI training, inference, and big data processing workloads through a single storage platform.

A key design principle of this architecture is the separation of metadata and user data.

JuiceFS provides the unified file system layer, while the metadata engine and Ceph RADOS independently handle metadata management and data storage. At the storage layer, Ceph distributes data and manages failure domains using the CRUSH algorithm, providing a decentralized and highly reliable storage backend.

The architecture also adopts a lightweight client model.

JuiceFS clients are responsible for file system semantics, metadata operations, cache management, and request scheduling. Once data is written to Ceph RADOS, data durability—whether implemented through replication or erasure coding—is handled entirely by the storage cluster.

This division of responsibilities keeps compute nodes lightweight, minimizes interference with AI training workloads, and fully utilizes the reliability and scalability already provided by the existing Ceph deployment.

For our private cloud environment, this modular approach significantly reduced engineering and operational overhead compared with developing and maintaining an integrated distributed storage system. At the same time, it provides unified access to AI training, AI inference, and big data workloads through a common storage platform.

Unified AI storage architecture

Our unified AI storage platform consists of four logical layers.

Workload layer

At the top are the application workloads, including virtual machines, Kubernetes-based containerized applications, traditional microservices, AI training, AI inference, and Apache Spark jobs.

Cache layer

The cache layer consists of two components:

DataCache Pool: It accelerates repeated access to training datasets, model files, and other hot data, reducing load on the storage backend.
KVCache Pool: It’s designed for large language model (LLM) inference. It caches context data across multiple storage tiers, reducing latency and improving the responsiveness of online inference services.

Storage protocol layer

JuiceFS provides a unified data access entry point here. AI training jobs access data via POSIX FUSE or CSI Driver; big data Spark jobs access data through the JuiceFS Hadoop Java SDK; and object storage toolchains access the same data via JuiceFS S3 Gateway. This approach unifies file storage, object storage, and big data access interfaces around a single underlying data set, reducing data movement and redundant storage across systems.

It should be noted that block storage scenarios, such as VM cloud disks, are primarily handled by Ceph RBD, which is part of the overall cloud storage system's block storage capability. The focus of this article is the file, object, and big data access foundation built on JuiceFS. While both share the underlying Ceph storage resources, they differ in access semantics and service targets.

Storage engine layer

JuiceFS uses a metadata-data decoupled architecture. For file systems with hundreds of millions of files, metadata capabilities directly impact path resolution, directory traversal, file attribute queries, small-file access, and concurrent job startup performance. TiKV provides distributed transactions, strong consistency, high availability, and horizontal scalability, offering stable metadata support for large-scale file systems. We built a five-node TiKV cluster for the metadata layer.

For the data layer, we use Ceph RADOS as the underlying data storage engine for JuiceFS. Ceph RADOS reuses existing private cloud storage resources, reducing duplicate infrastructure costs. In addition, by creating an erasure-coded pool for physical data storage, we improve physical disk capacity utilization while maintaining reliability, mitigating cost pressures from data growth.

In this architecture, data communication follows two paths:

The metadata path between clients and TiKV, handling path lookups, directory structures, file attributes, and state information
The data read/write path between clients and Ceph RADOS, handling actual data block access

Through librados, JuiceFS interacts directly with the underlying Ceph RADOS. Compared to traditional gateway-based storage architectures, this reduces multi-layer forwarding overhead, allowing clients to access the storage cluster more directly after metadata interactions and shortening the data access path.

Production deployment and optimization practices

Critical scenario deployment and storage consolidation

In LLM application scenarios, our company has dedicated self-built compute clusters for model inference services such as MiniMax. Meanwhile, custom models developed by internal application lines have gradually been migrated to JuiceFS after passing production validation. As a result, core LLM inference and training applications have begun to unify onto the new file storage foundation, with the original NFS clusters and Alluxio storage clusters entering phased decommissioning.

In core middleware and backup scenarios, including large-scale ClickHouse deployments, multiple core middleware systems and full production backup tasks have also been migrated to JuiceFS. As the unified file foundation has proven its capacity, stability, and access efficiency, MinIO and SeaweedFS have also entered decommissioning, reducing the operational complexity of maintaining multiple parallel storage systems.

Cache acceleration for AI training and inference

For AI training scenarios, training data typically has a "write once, read many" pattern. We use JuiceFS Community Edition's local data caching capability to cache hot training data on compute nodes. This reduces backend storage pressure and improves multi-node concurrent read efficiency.

For LLM inference scenarios, models serving online requests or performing iterative token generation are sensitive to context data loading latency and throughput. With limited single-node GPU memory and memory capacity, we use a multi-level caching mechanism to cache context data across multiple levels, improving online inference response efficiency.

Benchmarking and small I/O optimization

After deploying the unified storage platform, we focused on evaluating its performance under small-file workloads.

Using juicefs bench and fio, we benchmarked small-file operations and random I/O to compare the new JuiceFS + Ceph RADOS architecture with the previous JuiceFS + MinIO deployment.

To minimize the impact of caching, all local caches were disabled during testing, allowing I/O requests to be served directly by the storage backend. It's worth noting that these benchmarks were intended to validate the architecture rather than measure peak cluster performance. The tests were conducted with only four client threads under low concurrency.

Even under low-concurrency test conditions, the new architecture demonstrated substantial performance improvements.

The two tables below show test results for JuiceFS + MinIO:

Item	Throughput	Latency
Large file write	3,377.65 MiB/s	1.21 s/file
Large file read	4,248.77 MiB/s	0.96 s/file
Small file write	147.2 files/s	27.17 ms/file
Small file read	1,176.8 files/s	3.40 ms/file

Item	IOPS	Throughput (MiB/s)
Random read	57,504	1,483
Random write	1,194	1,521

The two tables below show test results for JuiceFS + Ceph RADOS:

Item	Throughput	Latency
Large file write	4,984.12 MiB/s	0.82 s/file
Large file read	4,817.87 MiB/s	0.85 s/file
Small file write	817.0 files/s	4.90 ms/file
Small file read	3,776.4 files/s	1.06 ms/file

Item	IOPS	Throughput (MiB/s)
Random read	77,984	3,394
Random write	3,836	4,120

Test result highlights:

Small-file write throughput improved by up to 5.5x, with significantly lower latency for individual file writes.
Small-file read throughput improved by up to 3.2x, delivering higher read throughput and lower per-file read latency.
Random I/O performance also improved. Both throughput and IOPS increased, with random-write IOPS improving by more than 3x.

From an architectural perspective, the JuiceFS + MinIO solution requires S3 interface and object storage gateway overhead, while the JuiceFS + Ceph RADOS solution uses librados for more direct interaction with the underlying storage cluster. This reduces gateway forwarding overhead. As a result, Ceph RADOS shows more significant performance advantages in scenarios sensitive to access paths, such as small-file writes and random writes.

However, in random small-file read tests, since local caching was disabled, data still needed to be retrieved from physical disks in real time. Therefore, read performance did not show the same order-of-magnitude improvement as writes, but still achieved a stable ~1.4x improvement overall.

Small-file space amplification and write amplification optimization

In erasure coding (EC) mode, traditional EC algorithms require alignment with stripe units. In a 6+3 configuration, even writing a 10-byte tiny file requires filling 6 data chunks to 4 KB each plus 3 coding chunks, resulting in 36 KB of physical storage—causing severe write amplification and thousands of times space waste.

To address this, we introduced Ceph's Fast EC feature to optimize small-object writes.

On one hand, Fast EC alleviates small-file space amplification. When enabled, small objects are no longer forced to fill entire stripes. For a 10-byte file, the logical size remains 10 bytes, while physical space occupies the first stripe; with a 4 KB minimum allocation unit and parity data, physical footprint is reduced to 16 KB.

On the other hand, Fast EC also improves read/write efficiency in small I/O scenarios. Reads no longer require fetching full stripes; writes can use parity delta writes (PDW) to reduce unnecessary network interactions and I/O overhead, improving overall small-file read/write performance.

JuiceFS Mount Pod stability optimization

In containerized environments, AI training applications occasionally encountered bad file descriptor errors, which severely impacted training efficiency. Investigation revealed that these errors were related to Mount Pod out-of-memory (OOM) conditions.

Periodic OOM due to metadata backups: When the cluster reached tens of millions or hundreds of millions of files, JuiceFS' default hourly metadata backups generated large backup files. Since these backup tasks were randomly assigned to clients in the cluster, and the underlying RADOS architecture imposes size limits on individual object writes (for example, 128 MB), backup tasks could fail repeatedly in large-scale file system scenarios, causing Mount Pod memory usage to grow continuously and eventually trigger OOM.

To address this, we disabled automatic backup on Mount Pods, preventing online clients from performing automatic backups. Instead, dedicated scheduled tasks handle metadata backups centrally, avoiding additional backup pressure on application clients.

Occasional OOM due to page cache growth: During frequent reads of large amounts of local cache, the Linux kernel page cache continued to grow. In such cases, even if the container's actual memory usage was only about 1.1GB, Kubernetes OOM decisions based on working_set_bytes—which includes page cache—could trigger OOM kills when the metric rose to high levels (for example, 33.5 GB), causing occasional client disconnections.

To address this, we increased limits.memory based on historical monitoring data. Then, we physically isolated cache resources, ensuring that different PVCs did not share the same local cache directory. For nodes with limited memory (such as CPU nodes), we capped local cache capacity (data cache size) within a reasonable range (for example, 10 GB to 50 GB) and enabled cache reclamation once the limit was reached. This prevents uncontrolled growth of the kernel page cache inside the container.

Through these optimizations, Mount Pod stability improved significantly in large-scale file system and high-frequency cache access scenarios. This provides a solid foundation for onboarding more AI training and inference workloads to the unified file foundation.

Future plans

We plan to focus on three aspects for the continued evolution of the AI cloud storage infrastructure:

Unified management from private cloud to hybrid cloud: As AI training jobs are scheduled across both private and public clouds, the storage foundation needs stronger cross-environment data management capabilities. Building on the existing private Ceph RADOS architecture, we’ll continue exploring JuiceFS' integration with public cloud object storage.

By pushing cross-cloud data synchronization, data distribution, and access path management down to the storage layer as much as possible, we can reduce the application's awareness of differences between cloud environments, providing more consistent mount paths and access experiences. This will improve training job migration efficiency across different compute environments and reduce data management complexity in multi-cloud settings.

Promoting lakehouse integration to reduce cross-system data movement: Previously, data movement between offline HDFS big data clusters and algorithm storage clusters relied on synchronization tools, increasing pipeline complexity, data redundancy, sync latency, and management costs.

We plan to further adopt and refine the JuiceFS Hadoop Java SDK so that Spark jobs in the algorithm layer can directly read and write the underlying unified storage pool. This will allow big data computing, algorithm training, and data archiving to operate around a single storage foundation, reducing cross-system data replication and redundant storage, further breaking down data silos, and advancing storage-compute separation and lakehouse integration.

Building tiered storage to accelerate LLM inference: As LLM inference services scale, a single storage medium can no longer simultaneously meet cost, capacity, and latency requirements. We plan to build capabilities in data tiering and KVCache tiering.
- For data tiering: JuiceFS' current data tiering feature is primarily designed for public cloud object storage. For private Ceph RADOS environments, we plan to collaborate with the community to explore mounting multiple Ceph storage pools of different performance tiers (for example, NVMe Pool and HDD Pool) under the same JuiceFS file system, allowing data with different access frequencies to reside on appropriate storage tiers.
- For KVCache tiering: As LLM inference evolves toward prefill/decode (PD) separation in distributed architectures, relying solely on single-node GPU memory to hold context data faces capacity and cost bottlenecks. Building on the unified storage foundation, we’ll gradually explore a four-tier KVCache architecture composed of HBM (GPU memory), DRAM, local SSD, and shared storage, providing more flexible context data management and access acceleration for large-scale AI inference clusters.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

Amazon S3 Files: How It Works, Performance Boundaries, and a Comparison with JuiceFS

DASWU — Fri, 24 Jul 2026 02:45:46 +0000

AWS announced Amazon S3 Files to let you mount an S3 bucket as a high‑performance shared file system on compute nodes without moving any data.

This isn't the industry's first attempt to make S3 accessible as a file system. From the early days of s3fs, to AWS' later release of Mountpoint for Amazon S3, and now S3 Files, the path to "using S3 like a file system" has been many years in the making. The key difference is that earlier solutions mostly operated at the access layer, while S3 Files finally wraps shared access, full file system semantics, and a managed high‑performance layer into a native offering.

This makes S3 Files a new option worth analyzing on its own. For workloads that need file‑style access to existing S3 data, it provides a native, lightweight solution. But in more complex scenarios such as AI model training or big data analytics, its actual performance still needs to be evaluated based on its underlying implementation and runtime behavior.

In this article, we’ll analyze S3 Files’ implementation, performance boundaries, and how it differs from JuiceFS.

S3 Files: an S3 native file system with EFS as the high‑performance layer

S3 Files uses Amazon Elastic File System (EFS) as a managed high‑performance storage layer to handle low‑latency data and metadata. On top of that, it provides full file system semantics for S3, including consistency, file locking, and POSIX permissions.

Think of it this way: AWS adds an EFS-based file system access layer on top of object storage, so that data previously accessible only through object APIs can be used directly by compute nodes as directories, files, and mount points. Changes between the file system and S3 are synchronized automatically by the service in the background.

With this architecture, S3 Files does not migrate the entire dataset. Instead, it only places a portion of the current working set into the high‑performance layer on demand, while the “source of truth” for the data remains in S3.

How S3 Files works: mount, import, and sync

For S3 Files, mounting is only the beginning. What really matters is the data path after mounting: how the scope is determined, what gets imported on first access, which requests go to the high‑performance layer, and how writes are synced back to S3. These mechanisms directly determine the performance boundaries and cost structure discussed later.

The figure below shows S3 Files mount architecture:

Taking an EC2 instance mounting an existing S3 bucket as an example, the real questions are not about the mount command itself, but about what happens after mounting: how data is imported, accessed, and synced. Here are the key technical details.

Scope: import the entire bucket or a specific prefix

In S3, files are organized using prefixes. You can think of them as similar to “subdirectories” or “sub-buckets” within a bucket, but they are not actual directories. S3 Files can mount an entire S3 bucket (i.e., s3://my-bucket/) or restrict the scope to a specific prefix (i.e., only s3://my-bucket/data/ml/). This is especially important for huge S3 buckets containing millions or even hundreds of millions of objects, because a broader scope adds more metadata sync overhead and potentially more operational complexity.

When using S3 Files on a compute node, AWS provides a custom mount client amazon-efs-utils. When mounting, you use the file system ID assigned by AWS for the S3 Files instance, not the bucket name.

You can create a local mount directory and mount it using the dedicated s3files file system type as shown below:

sudo yum -y install amazon-efs-utils
sudo mkdir /mnt/s3files
sudo mount -t s3files file-system-id:/ /mnt/s3files

When mounting, you can specify a subdirectory (or actually a prefix) in the mount path (for example, file-system-id:/data/ml/). However, this only affects the local mount point, as the underlying metadata sync still applies to the entire bucket, and other mount points could inadvertently access other subdirectories. Very often in practice, a better approach is to limit the scope to a specific prefix when creating the S3 Files instance, rather than using the entire bucket and trying to restrict access afterward.

First access: import triggers and size thresholds

S3 Files does not immediately move the entire dataset into the high‑performance layer after mounting. Data import is triggered by access events. The default mode is trigger=ON_DIRECTORY_FIRST_ACCESS: when you first access a directory, the system imports the metadata of files under that directory and asynchronously moves the data of small files that meet the criteria into the EFS high‑performance layer.

If configured as trigger=ON_FILE_ACCESS, the first directory traversal only imports metadata; data enters the high‑performance layer only when a file is actually read for the first time. This saves space and import cost, but first‑read latency is higher.

The most critical control parameter here is sizeLessThan, which takes a value in bytes. By default, only files smaller than 128 KiB (i.e., sizeLessThan=131072) are moved into the high‑performance layer during import. Larger files only have their metadata imported; their content is still retrieved mainly from S3. In other words, S3 Files optimizes primarily for small files and low‑latency access, not for warming up all data into the high‑performance layer. For AI training datasets that consist mainly of 10 MB‑level images or video files, this is crucial: even after a directory traversal, those large files may not actually enter the high‑performance layer under default settings.

Sync intervals and conflict resolution

S3 Files automatically maintains bidirectional sync between the file system and S3 in the background. When data in S3 changes, the file system view updates accordingly. Writes from compute nodes first land in the EFS high‑performance layer and then are batch‑synced back to S3 asynchronously. By default, the system aggregates modifications for a period before writing back.

The conflict resolution principle is clear: S3 is always the source of truth. If a file system modification hasn’t yet been synced back and the corresponding object has been updated by another application in S3, the system uses the latest version from S3 and moves the conflicting file to a .s3files-lost+found-* directory.

Performance boundaries and cost structure of S3 Files

The previous section explained how S3 Files works. This section discusses the resulting performance boundaries and cost structure. High‑performance layer occupancy, large‑file read path, write flow, and the amplification effects of partial updates and directory operations are the four most important aspects to evaluate.

Occupancy, eviction, and cost of the EFS high‑performance layer

S3 Files does not use capacity‑based LRU eviction. Instead, it uses access‑time‑based lifecycle management. By default, data that has been synced to S3 and not read for 30 days is evicted from the EFS high‑performance layer. This period is controlled by daysAfterLastAccess (configurable from 1 to 365 days).

This means the cost depends on how much data needs to reside in EFS and for how long. If your working set is large and stays active for a long time, costs will rise accordingly.

Large‑file direct reads and random reads: essentially client passthrough

S3 Files does not route all large‑file reads through the EFS high‑performance layer. With the default sizeLessThan of 128 KiB, only files below that threshold have their data moved into the high‑performance layer during import. For files already synced to S3, reads of 128 KiB or larger are streamed directly from S3.

In other words, S3 Files focuses on optimizing small‑file access and low latency, not on keeping large‑file reads in the high‑performance layer.

This direct‑read path requires that the compute resources themselves have permission to read the source bucket. AWS documentation explicitly requires roles to have s3:GetObject and s3:GetObjectVersion permissions; otherwise, the client cannot read directly from S3.

Cost of sequential writes: large‑scale writes introduce additional flow costs

The write path of S3 Files is not directly to S3. All writes first go into the EFS high‑performance layer and then are asynchronously synced back to S3.

This means that if your workload continuously generates large amounts of result data – for example, sequentially writing hundreds of terabytes of training outputs or analysis results – that data will incur two extra costs when flowing through S3 Files:

Data flow cost: Writes go into the high‑performance layer, then are later synced to S3. Compared to writing directly to S3, this path has an additional intermediate flow overhead.
Short‑term residency cost: After data is synced, it’s not immediately evicted from the high‑performance layer. It stays until the eviction condition is met. So temporary data from large writes may occupy EFS capacity for some time.

Using the current AWS US East (N. Virginia) regional pricing as an example: writing to EFS is about $0.06/GB, and reading it back for sync to S3 is about $0.03/GB. For the data flow alone, each 1 TB written adds roughly $90 in extra cost. If that data continues to reside in EFS after sync, further high‑performance layer storage costs will apply.

This is why S3 Files is better suited for reading existing data than for sustaining large‑scale, continuous result writing.

Partial updates and directory operations: amplification due to the object model

S3 Files does not chunk data at the file system level. It tries to maintain a direct mapping between files and S3 objects. The drawback: when you perform a small random write or append to a large file – from the application’s perspective a tiny update – the backend sync back to S3 can easily amplify into a full object write and versioning overhead.

For example, a user appends a 100 KB image key to a 100 GB LMDB file through S3 Files. The application sees a small write. But this modification is not immediately written back to S3. It’s aggregated for up to 60 seconds and then synced to the bucket. Unlike block storage, which would only update a single block, this type of update is more likely to be amplified into an object write, sync latency, and version storage costs. The larger the file and the more frequent the modifications, the more serious this issue becomes.

Directory renaming is similarly constrained by S3’s flat namespace. S3 does not have native directory metadata. Therefore, when you perform rename or mv, S3 Files cannot just update a single metadata entry. Instead, it must write new objects for every file under the directory and delete the old ones. For a directory with tens of millions of objects, this significantly prolongs sync time and increases S3 request costs. Until the sync completes, the file system view and the S3 view may be temporarily inconsistent.

Overall, S3 Files’ strengths are native access, zero data migration, and good compatibility with existing S3 assets. The drawback, however, is that once the workload shifts to large-file reads, continuous writes, frequent partial updates, or operations on large directories, both performance and cost are amplified more quickly. For this reason, S3 Files is better suited for lightweight shared access scenarios, while in heavy workloads such as training, data production, and large-scale analytics, its drawbacks tend to emerge earlier.

JuiceFS vs. S3 Files: two different architectural philosophies

As we saw earlier, many of S3 Files’ boundaries are not accidental; they are typical of solutions that try to map files directly to S3 objects. Whether it’s the early s3fs, Mountpoint for Amazon S3 (which focuses on high‑throughput reads), or today’s S3 Files, all of them try to maintain a direct mapping between files and S3 objects in exchange for transparent access to existing S3 data.

The advantage of this approach is transparency and low‑effort adoption. The cost has inherent limitations due to the S3 object model. That’s why directory operations tend to devolve into per‑object requests, and partial updates on large files easily amplify into write amplification, sync delays, and extra costs.

This is precisely why the difference between JuiceFS and these solutions is not a matter of a single feature or metric, but a fundamental divergence in architectural philosophy. JuiceFS is not an access layer that “mounts S3 as a file system”; it’s a cloud‑native distributed file system built on top of object storage. It adopts a decoupled metadata and data architecture: file data is stored in the underlying object storage, while metadata is independently managed by a high‑performance key‑value store. This makes JuiceFS much better suited for heavier production workloads such as training, analytics, and data production.

To help you compare these two architectures, here is a comprehensive table comparing JuiceFS and S3 Files:

Comparison basis	JuiceFS	Amazon S3 Files
Architecture	Metadata and data decoupled; files are chunked and written to object storage	EFS‑based proxy; data is not chunked; 1:1 mapping based on file size (< 128 KB by default)
Core cost	Open source software; costs from object storage, metadata engine, and cache resources	In addition to S3 storage, pay for EFS storage ($ 0.30/GB) plus data sync fees
Read/Write amplification (random writes)	Very low in many cases. Because of chunking, a partial random write typically updates only the affected block, not the whole file	Can be very high in data production scenarios. Random modification of a large file leads to rewriting and retransmitting the entire object
Tiering strategy	Based on capacity and access heat; hot data is automatically warmed up into local disk/memory cache on compute nodes	Based on file size and access time. Small files (< 128 KB) are cached in EFS; large files bypass EFS and read/write directly to S3
Small‑file performance	Relies on a fully in‑memory metadata engine (like Redis or TiKV); well suited for many small files and metadata operations	Depends on EFS performance and the NFS protocol
Large‑file throughput	Can use local NVMe / memory cache to boost throughput	Depends on EFS gateway or S3 direct performance; large‑scale parallel throughput is tied to capacity quotas
Cache consistency	Strong consistency (close‑to‑open) enforced by the independent metadata service	NFS close‑to‑open. But when concurrent modifications conflict between S3 and the file system, local EFS data is discarded into lost+found and S3 is forced as the source of truth
POSIX compatibility	Nearly 100% compatible. Supports hard links, atomic rename, and full lock semantics	Subset of NFSv4.1/4.2. Does not support hard links or atomic rename
Permission management	Supports standard POSIX permissions, ACLs, extended ACLs, and more	Supports standard POSIX permissions, ACLs, extended ACLs, and more
Encryption and security	In‑transit encryption and at‑rest encryption	In‑transit TLS encryption and at‑rest SSE‑KMS encryption
AI‑specific optimizations	Deeply optimized for AI‑common formats (such as LMDB and Safetensors) with mmap reads and local data warm-up	No AI‑specific optimizations; relies on basic streaming reads

Summary

There is no silver bullet. The right choice depends on your specific scenario.

The launch of S3 Files fills a gap in the AWS official ecosystem: seamless, zero‑migration native conversion of S3 into a file system. Its design philosophy is clear: 100% format transparency with the S3 object ecosystem, with targeted optimizations for small‑file (<128 KB) read/write performance in AI scenarios.

When to choose S3 Files

If your core need is to allow legacy applications, shell scripts, or traditional software to access existing S3 data as files without changing your architecture, or if you need a general shared file space that is primarily used for read‑only, small‑file, sequential workloads, then S3 Files is the more natural choice. Its native managed service, plug‑and‑play, and zero data migration can significantly lower the barrier to entry (though you may need to pay high EFS storage and sync costs for that convenience).

When to choose JuiceFS

If your workloads involve AI model training, data production, high‑performance computing (HPC), or big data analytics – facing tens of millions of small files, random reads/writes of terabyte‑scale large files, or demanding higher mmap performance, cache hit rates, and overall throughput – then JuiceFS is the better fit. Compared to S3 Files, JuiceFS’ data chunking, independent metadata engine, and more flexible caching system make it better suited for heavy‑workload, long‑running production and AI training/inference file system scenarios.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

JuiceFS 1.4: Lower Costs, Faster Metadata, and Better Control for Massive Data Management

DASWU — Wed, 15 Jul 2026 09:04:24 +0000

JuiceFS Community Edition 1.4 is released. It’s the fifth major release since the open source edition was introduced in 2021 and is now the new Long-Term Support (LTS) version. We’ll continue to maintain both v1.4 and v1.3, while v1.2 has reached end of maintenance.

JuiceFS has now surpassed 14.2k GitHub stars. According to anonymous usage statistics reported by users, the total amount of data managed by JuiceFS Community Edition has exceeded 1.4 EB, representing more than 700× growth since 2022.

As JuiceFS is increasingly adopted for large-scale data management, high-concurrency workloads, and multi-user shared environments, long-standing challenges such as storage cost optimization, metadata performance, and resource governance have become more prominent. These are the primary focus areas of the 1.4 release.

In this post, we'll walk through the key improvements in JuiceFS 1.4, including tiered storage, faster metadata operations, enhanced resource management, more reliable data synchronization, metadata change tracking, and broader platform compatibility.

Lower storage costs with file- and directory-level tiered storage

As file systems continue to grow, different datasets naturally diverge in access frequency, performance requirements, and retention periods. Using a single storage type uniformly makes it difficult to simultaneously meet the performance needs of frequently accessed data and the cost-control requirements of infrequently accessed data. Object storage typically offers different storage classes based on access patterns, including hot data, warm (infrequent access) data, and cold (archival) data storage.

JuiceFS has supported setting object storage types via --storage-class since v1.1, but the configuration granularity was mainly at the file system default or mount point level. JuiceFS 1.4 integrates storage class into the file system semantics, supporting storage tier settings on a per-file or per-directory basis. Directory-level configurations can be inherited by subsequently created files and subdirectories. This facilitates tiered management by project, dataset, or application directory.

The storage tiers can be configured flexibly according to the object storage vendor being used. When writing new data, JuiceFS writes it to the corresponding object storage type based on the configuration of the file or its parent directory. For existing data, you can also adjust the metadata configuration and leverage the data migration capabilities on the object storage side to move it to a new storage tier. This capability is suitable for scenarios such as AI training datasets, log archiving, backup data, historical experiment data, and offline analysis results. For archive storage, it’s still necessary to evaluate retrieval latency and fees. For more implementation details, usage methods, and future evolution, see A Deep Dive into JuiceFS 1.4 Tiered Storage.

Faster metadata operations: batch delete, batch clone, and hotspot read optimization

In workloads involving massive numbers of small files, large directories, or high-concurrency access, metadata operations often become the primary performance bottleneck.

JuiceFS 1.4 addresses write transaction overhead and hotspot read overhead in metadata operations with optimizations including batch delete, batch clone, and Redis client-side caching.

Batch delete and clone: reducing transaction overhead

Previously, deleting a large number of files required the system to process them one by one, sequentially updating directory entries, inodes, space statistics, trash, and quota metadata. JuiceFS 1.4 consolidates the deletion of multiple non-directory files within the same directory into a batch transaction, reducing the repetitive overhead of per-file operations. This is applicable to scenarios like large directory cleanup, temporary data reclamation, training sample cleanup, and log directory deletion.

Batch cloning targets directory replication and snapshot scenarios. juicefs clone does not copy the underlying data blocks but creates new file records at the metadata layer and reuses the source file's data block references. JuiceFS 1.4 further reduces the metadata transactions generated by per-file cloning by processing clones of multiple files within the same directory in batch. This is ideal for AI dataset version management, experiment environment preparation, and large-scale directory snapshots.

Redis client-side caching: reducing hotspot metadata read overhead

In high-concurrency reads, path resolution, directory entry lookups, and file attribute queries generate a large number of repeated requests. When Redis is used as the metadata engine, these requests require round trips between the client and Redis. This may impact access latency and increase Redis load.

JuiceFS 1.4 caches hot inode attributes and directory entries locally on the client side. When cached, it can reduce repeated queries to Redis. When related metadata changes, the local state is updated through a cache invalidation mechanism. It's important to note that this capability caches metadata, not file content.

It’s particularly beneficial for read-heavy workloads with stable hot paths, such as AI training data loading, large-scale container startup, and multi-task concurrent reading. For more implementation details, see Faster Metadata Operations with Batch Unlink, Batch Clone, and Redis Client-Side Caching.

Improved resource management: user quotas and trash usage statistics

In distributed storage environments, storage resources are often shared among multiple users, teams, and projects.

Without effective governance, accidental writes or abnormal workloads from a single user can quickly consume large amounts of storage space or inodes. This affects both system stability and operating costs. Quota management is a critical means of establishing predictable resource boundaries in shared environments.

JuiceFS Community Edition 1.4 introduces user and group quotas, allowing administrators to monitor, configure, and enforce resource limits based on identities.

Resource governance is now extended beyond file system and directory quotas to include user- and group-level quotas, making it especially suitable for shared clusters and AI training platforms.

To reduce metadata overhead in multi-client environments, JuiceFS uses asynchronous accounting so that usage statistics converge gradually over time. For details, see Quota Design in Distributed Architectures: Implementation and Use Cases in JuiceFS.

Supported quota types include:

Quota type	Scope	Design goal	Typical use case
Total file system quota	Entire file system	Prevents overall resource runaway	Cost budget control, capacity limit
Subdirectory quota	Directory subtree	Blocks abnormal write behavior	Prevents misoperations, small‑file storms
User quota	Per user	Isolates impact between different applications	Multi‑tenant data management
User group quota	Project or department	Cost allocation and team limits	Shared environment for AI projects

JuiceFS 1.4 also improves trash space visibility.

Deleted files may remain in the trash for a retention period, making it difficult to understand why storage space has not yet been reclaimed. The enhanced summary tool now reports trash usage, helping administrators identify storage consumption and make informed cleanup, retention, or expansion decisions.

Expanded capabilities: sync, backup, and change tracking

More reliable large-scale sync

Large-scale migration, cross-cloud sync, backup, and archival workloads often face interruptions, security requirements, and bandwidth contention.

JuiceFS 1.4 significantly enhances juicefs sync with three major capabilities:

Resumable sync: It reduces recovery costs after task interruptions. During synchronization, JuiceFS records the task progress. If the task exits abnormally or is manually interrupted, it can resume from the saved state, reducing repeated scanning and processing. This capability is suitable for migration and backup scenarios with a large number of objects, long task durations, or unstable cross-cloud links.
Data encryption and decryption: In cross-cloud backup and archiving scenarios, client-side encryption is a common compliance requirement. JuiceFS 1.4 supports completing encrypted writes, decryption recovery, or re-encryption within the sync pipeline, reducing reliance on external encryption tools. This capability is suitable for off-site backup, sensitive data migration, key rotation, and compliance auditing. However, it requires careful management of key storage and recovery processes.
Global traffic control: It provides bandwidth constraints for concurrent multiple sync tasks. Compared to per-process rate limiting, version 1.4 can centrally manage the overall bandwidth usage of multiple sync tasks, reducing the impact of sync tasks on online application and other network activities. This is suitable for cross-cloud transfers, multi-task concurrent backups, data center migrations, and shared outbound link scenarios. For implementation details, see JuiceFS Sync for PB-Scale Data Transfers: Resumable Sync, Encryption, and Bandwidth Control.

Changelog: metadata changes become traceable

JuiceFS Community Edition 1.4 introduces a metadata changelog capability, which records metadata change events across the file system.

Previously, troubleshooting relied primarily on client-side access logs, which only reflected operations performed through individual mount points. In multi-client deployments, reconstructing a complete sequence of events was often difficult.

A changelog records metadata operations—including file creation, deletion, attribute updates, and renames—directly at the metadata layer, providing a unified source for troubleshooting, auditing, and incremental processing.

Administrators can now quickly identify accidental deletions, unexpected renames, permission changes, and metadata modifications without collecting logs from every client.

When issues like accidental deletion, abnormal renaming, or unexpected permission or attribute changes occur, administrators can review the relevant change records based on the changelog. This reduces dependence on single-client logs and shortens the troubleshooting path. It also provides a more unified source of metadata changes for operational auditing.

In backup, migration, and recovery scenarios, the changelog can serve as a reference for incremental processing. For large-scale file systems, numerous changes may occur between two full backups or migration tasks. By recording the metadata changes during this period, the changelog can provide input for subsequent incremental backups, migrations, or recovery processes, reducing reliance on full scans.

Better support across different environments

JuiceFS Community Edition 1.4 further improves compatibility across diverse deployment environments.

On Windows clients, the release improves cross-platform consistency and stability, including user mapping, permission mapping, and file access behavior, reducing compatibility issues when Linux and Windows clients access the same file system.

For the Java SDK and the Hadoop ecosystem, JuiceFS 1.4 adds Kerberos authentication, completing support for Hadoop secure mode.

JuiceFS 1.3 already introduced Apache Ranger integration for authorization and access control. Together, Kerberos authenticates who the user is, while Ranger determines what the user can access, providing a more complete security model for enterprise big data platforms.

On the storage backend side, JuiceFS 1.4 also adds support for SMB/CIFS-based storage. This makes it easier to integrate with existing NAS or file-sharing infrastructure.

Continued growth in scale and AI adoption

According to anonymous usage statistics, JuiceFS Community Edition now powers nearly 70,000 file systems managing over 1.4 EB of data, with deployment scale continuing to grow.

Over the past year, AI applications have continued to expand from model training to inference services, agents, and multi-cloud scheduling, placing higher demands on data storage. These changes are also reflected in the use cases shared by community users, covering areas such as large language models, autonomous driving, quantitative investment, and computing platforms. We thank these users for sharing their real-world practices, providing valuable references for more teams building AI data infrastructure.

New user stories:

AI training and large language models

AIGC

Autonomous driving and robotics

Inference and AI agents

Quant investment

JuiceFS+MinIO: Ariste AI Achieved 3x Faster I/O and Cut Storage Costs by 40%+

Big data

NAVER, Korea's No.1 Search Engine, Chose JuiceFS over Alluxio for AI Storage

Development of JuiceFS 1.4 spanned nearly a year. During the release cycle, the community reported 366 issues, merged 515 pull requests, and welcomed contributions from 59 contributors. We sincerely thank everyone who reported issues, contributed code, improved documentation, and helped evolve JuiceFS for increasingly demanding production environments. Your participation drives the rapid growth of JuiceFS.

Download and try JuiceFS 1.4 here.

JuiceFS Sync for PB-Scale Data Transfers: Resumable Sync, Encryption, and Bandwidth Control

DASWU — Fri, 10 Jul 2026 06:56:21 +0000

In scenarios such as data migration, cross-cloud synchronization, and object storage backup, juicefs sync is commonly used to transfer large volumes of data. When datasets grow to the TB- or PB-scale, with millions or even billions of objects, a single synchronization task may run for hours or even days.

As these long-running jobs progress, several common challenges tend to emerge:

After network interruptions, process crashes, or node restarts, tasks often struggle to resume from a consistent state and may need to rescan or reprocess data.
Backup workflows may expose plaintext data and face compliance or security requirements.
When multiple sync jobs run concurrently, bandwidth contention becomes significant, while the overall transfer process lacks effective global control.

To address these challenges, JuiceFS 1.4 introduces three major enhancements to sync: resumable sync, data encryption/decryption, and global traffic control.

In this article, we’ll explain the use cases, implementation details, and configuration methods for each feature.

Resumable sync

In earlier versions, if a synchronization task failed or was interrupted, rerunning juicefs sync required rescanning both the source and destination before determining which objects had already been synchronized and which still needed to be copied.

For workloads involving hundreds of millions of objects or large files, the scan itself could incur substantial time and object-storage request costs.

To address this issue, JuiceFS 1.4 introduces a resumable sync mechanism for sync. When enabled, synchronization progress is periodically saved to the destination. If the task is interrupted, rerunning the same command automatically locates and loads the matching checkpoint and resumes from the last unfinished position, avoiding a full restart.

How it works

When resumable sync is enabled, sync stores a JSON state file on the destination side:

.juicefs-sync-checkpoint.<hash>.json

The <hash> value is derived from the source, destination, and key synchronization parameters. This ensures that a task only loads checkpoints created for itself, preventing accidental reuse across different jobs.

The workflow is shown below:

Checkpoint save, restore, and cleanup workflow in juicefs sync:

When sync starts, it first looks for a checkpoint matching the current task.
If a matching checkpoint is found, execution resumes from the saved state. Otherwise, synchronization starts normally with a fresh scan. sync traverses multiple prefixes concurrently, maintaining independent state for each prefix, including:

- Whether traversal is complete  
- The last scanned position  
- Pending objects to synchronize  
- Failed objects

When restoring from a checkpoint:

- Pending and failed objects recorded in the checkpoint are re-added to the task queue.  
- Prefixes that were not fully traversed resume scanning from their saved positions.  
- Fully traversed prefixes only continue processing unfinished objects recorded in the checkpoint.

During execution, progress is saved asynchronously at a configurable interval, which defaults to every 10 seconds.
After successful completion, the checkpoint file is automatically removed. If the task fails or is interrupted, the checkpoint is retained for resumption on the next execution of the same command.

In cluster mode, only a single checkpoint exists and is maintained centrally by Manager.

Workers do not directly read or write checkpoint files on the destination. Instead, they:

Pull tasks from Manager
Execute synchronization
Report results back to Manager

Manager aggregates completed objects, failed objects, statistics, and multipart-upload state into the global checkpoint.

Usage

# Enable resumable sync.
juicefs sync --enable-checkpoint SRC DST

# Customize checkpoint save interval (default: 10s).
juicefs sync --enable-checkpoint --checkpoint-interval 30s SRC DST

# Ignore existing checkpoints and restart from scratch.
juicefs sync --enable-checkpoint --checkpoint-force-reset SRC DST

Data encryption and decryption

For cross-cloud backup and archival workflows, client-side encryption is often required to satisfy compliance requirements such as data sovereignty, encryption at rest, and secure migration of sensitive data.

Previously, juicefs sync did not provide built-in encryption capabilities. Users who wanted to write encrypted data to the destination typically had to use external tools for additional processing.

In JuiceFS 1.4, streaming encryption and decryption are integrated directly into the synchronization pipeline, enabling three common workflows:

Encrypt-on-write: Encrypt plaintext data before writing it to the destination, suitable for encrypted backup and archiving.
Decrypt-on-read: Read encrypted data from the source and write decrypted data to the destination, suitable for data recovery or plaintext migration.
Re-encryption: Decrypts source data with an old key and re-encrypts it with a new key before writing to the destination, suitable for key rotation or cryptographic algorithm migration.

Chunk-based streaming encryption

To support object storage Range GET operations while avoiding excessive memory usage for large files all at once, sync uses a fixed-size 1 MiB chunk-based streaming encryption scheme.

A file is first divided into plaintext chunks:

[chunk 1: 1 MiB][chunk 2: 1 MiB] ... [chunk N: ≤1 MiB]

Each plaintext chunk is encrypted independently.

Each encrypted chunk consists of a 4-byte header and the ciphertext data, where the 4-byte header stores the actual ciphertext length (ct_len):

Each encrypted block: [4B ct_len][ciphertext + padding]

Encrypted file: [encrypted chunk 1][encrypted chunk 2] ... [encrypted chunk N]

The encrypted block size is determined by the plaintext chunk size plus encryption overhead: plainChunkSize + overhead. The plainChunkSize is fixed at 1 MiB, and the overhead depends on the encryption algorithm and key type used.

This design allows random reads to retrieve only the required encrypted chunk rather than downloading the entire file. Because encrypted objects contain additional headers, padding, and encryption metadata, the destination object is typically larger than the original plaintext file.

Supported algorithms

The table below shows the supported algorithms:

Option	Symmetric cipher	Key encapsulation	Typical use case
aes256gcm-rsa (default)	AES-256-GCM	RSA	General-purpose workloads
chacha20-rsa	ChaCha20-Poly1305	RSA	Environments without efficient AES hardware acceleration
sm4gcm	SM4-GCM	SM2	Scenarios requiring Chinese commercial cryptography standards

Usage

The following examples use RSA keys.

Generate a key pair:

# Generate an RSA private key (the public key is derived automatically).
openssl genrsa -out private.pem 2048

# Generate a password-protected private key.
openssl genrsa -aes256 -out private.pem 2048

Scenario 1: Encrypt and write to destination

juicefs sync /local/data s3://mybucket/backup 
    --encrypt-rsa-key /path/to/private.pem

Scenario 2: Decrypt and read from source for data recovery or plaintext migration

juicefs sync s3://mybucket/backup /local/data 
    --decrypt-rsa-key /path/to/private.pem

Scenario 3: Re-encrypt for key rotation or algorithm migration

# Decrypt data encrypted with the old key and re-encrypt with the new key to new storage.
juicefs sync s3://old-bucket/encrypted s3://new-bucket/re-encrypted 
    --decrypt-rsa-key /path/to/old-private.pem 
    --encrypt-rsa-key /path/to/new-private.pem

If the private key is password-protected, the password can be provided via environment variables:

# For encryption scenarios, use JFS_ENCRYPT_RSA_PASSPHRASE.
export JFS_ENCRYPT_RSA_PASSPHRASE="your-passphrase"
juicefs sync /local/data s3://mybucket/backup --encrypt-rsa-key private.pem

# For decryption scenarios, use JFS_DECRYPT_RSA_PASSPHRASE.
export JFS_DECRYPT_RSA_PASSPHRASE="your-passphrase"
juicefs sync s3://mybucket/backup /local/data --decrypt-rsa-key private.pem

Notes:

Encrypted data is stored using a JuiceFS-specific format and can only be decrypted through juicefs sync with the corresponding key.
Back up encryption keys carefully. Once a private key is lost, encrypted data cannot be recovered.

Global traffic control

In earlier versions, juicefs sync already supported per-process rate limiting via --bwlimit. However, when multiple sync processes run concurrently—such as multiple Workers in a distributed sync, or multiple independent sync tasks sharing the same egress link—per-process limiting cannot constrain total bandwidth usage. The egress link may still be saturated, affecting other application traffic.

JuiceFS 1.4 introduces the --traffic-control-url parameter. Multiple sync processes can connect to the same external traffic control service, which allocates bandwidth quotas uniformly, enabling cross-process, cross-task global rate limiting.

How it works

Global traffic control uses a token bucket model. Before transmitting data, each sync process requests byte credits from the same traffic-control service.

Each process periodically requests a certain number of bytes (credit) before data transfer.

The traffic-control service determines:

How many bytes to grant
How long the granted quota remains valid

When credits are exhausted, the process requests additional credits.

If a quota is about to expire before being fully consumed, the unused portion is returned to the service in advance.

The service exposes a simple HTTP API for granting and reclaiming quotas. This must be implemented by the user or integrated with an existing service:

POST /traffic-control
Content-Type: application/json

Request:
{"bytes": 1048576}
  bytes > 0: Request byte credits.
  bytes < 0: Return unused credits.


Response:
{"granted": 524288, "expired": 1000}
  granted: Number of bytes granted this time.
  expired: Credit validity period (milliseconds).

During synchronization, sync requests quotas from the traffic control service before transmitting data. If no credits are available, transmission blocks until new credits are obtained. In this way, multiple sync tasks can share a single global bandwidth limit, preventing the total traffic from becoming uncontrolled even when individual tasks have their own limits.

Usage

# Deploy a traffic-control service first.
# (Example: listen on port 8080 and cap total bandwidth at 100 Mbps)
# (Service implementation is user-defined; JuiceFS only calls the API)

# Multiple sync processes share the same control service.
juicefs sync SRC1 DST1 --traffic-control-url http://127.0.0.1:8080/traffic-control &
juicefs sync SRC2 DST2 --traffic-control-url http://127.0.0.1:8080/traffic-control &

--traffic-control-url can be combined with --bwlimit.

The two mechanisms are independent:

--bwlimit limits the bandwidth of a single sync process.
--traffic-control-url limits aggregate bandwidth across multiple processes.

# Per-process limit: 50 Mbps. All processes combined respect the service-side cap.
juicefs sync SRC DST 
    --bwlimit 50 
    --traffic-control-url http://controller:8080/traffic-control

Summary

JuiceFS 1.4 enhancements to sync include:

Resumable sync reduces recovery costs after task interruptions.
Encryption and decryption improve the security of backups and archival data.
Global traffic control enables multiple synchronization tasks to share bandwidth in a coordinated manner.

For scenarios such as data migration, cross-cloud sync, object storage backup, and encrypted archiving, users can combine these capabilities flexibly based on task scale, network environment, and security requirements.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

Monitoring JuiceFS with Better Stack

DASWU — Fri, 26 Jun 2026 07:46:14 +0000

After deployment, JuiceFS feels like a local drive, but underneath it's a sophisticated distributed system. This perfectly reflects one of its core design principles: distributed systems are complex, but from a user's perspective, they should be simple to use.

Even so, that simplicity on the surface doesn't negate the need for deep visibility. For any critical storage system, gaining real-time visibility into its operations is crucial to prevent subtle performance degradations from escalating into significant incidents.

Fortunately, JuiceFS exposes a suite of monitoring metrics, including throughput, IOPS, latency, data size, and many more, in the widely adopted Prometheus format, making it ready for modern monitoring stacks. Traditionally, you would probably pair Prometheus with Grafana to collect these metrics and visualize them. This is indeed a powerful combination. However, deploying, managing, and maintaining these systems yourself adds operational overhead again. Ironically, you may want to monitor them too, and trust me, you would rather not create yet another monitoring stack just to monitor your Prometheus and Grafana combo.

That's where Better Stack comes in. It is a fully managed SaaS observability platform that combines user-friendly dashboards, tracing, logging, error tracking, incident management, automatic alerting, and even AI-powered SRE, all for a predictable, cost-effective price. With Better Stack, you get the power of the best-in-class tools out of the box without the operational overhead.

In this post, we'll guide you through setting up a comprehensive monitoring system for JuiceFS using Better Stack, from metric ingestion to intelligent alerting, so you can ensure your file system remains healthy and performant.

Preparing the JuiceFS file system

Before diving into setting up Better Stack for monitoring, you'll need an existing JuiceFS file system that is actively publishing metrics. JuiceFS Community Edition and JuiceFS Enterprise Edition (our cloud service is based on JuiceFS Enterprise Edition) both expose real-time status metrics in Prometheus format, but they do it in slightly different ways.

For the JuiceFS Community Edition, after mounting the file system, JuiceFS automatically exposes metrics via http://localhost:9567/metrics by default on the mounting host where the JuiceFS client is running. You can customize this port using the --metrics option if needed.

On the other hand, for JuiceFS Enterprise Edition & Cloud Service, metrics are exposed through the console via dedicated API endpoints. You'll need to replace VOLUME_NAME with your file system name and API_TOKEN with your API token. In this case, both Prometheus and JSON formats are available for metrics:

Prometheus: https://juicefs.com/api/vol/VOLUME_NAME/metrics?token=API_TOKEN
JSON: https://juicefs.com/api/volume/VOLUME_NAME/status?token=YOUR_TOKEN

A quick but important note: metrics are only generated when the file system is mounted. So before proceeding, ensure your JuiceFS file system is properly mounted and accessible. In this guide, we will use the JuiceFS Cloud Service, as it's the simplest to get started. If you haven't set up JuiceFS yet, please refer to the documentation for detailed instructions. Once you have created the first file system, URLs for the metrics mentioned above would be available under its Monitor tab.

Setting up a metrics source in Better Stack

With your JuiceFS file system up and running (don't forget to mount the file system to a host machine) and publishing metrics, the next step is to configure Better Stack to start ingesting that data.

First, if you haven't already, register for a Better Stack account. The process is seamless. Using a work email is recommended, and the platform provides clear guidance to help you set up your account and organization.

Once you're logged in, follow these steps:

In the left-hand navigation panel, head to Telemetry.
Under the Sources section, click Connect source.
Give your telemetry data source a descriptive name, such as "jfs-better-stack" or "juicefs-production", to easily identify it later.

Now, you'll configure how Better Stack should collect your metrics. In the collector settings:

Under Metrics, choose the Prometheus scrape option and click Connect source.
In the URLs to scrape section, input the JuiceFS metrics endpoint as described above.

Note that if you are not using the JuiceFS Cloud Service and your JuiceFS endpoint is behind a firewall, you'll need to allow traffic from Better Stack's scrape servers. The list of IP addresses to add to the allowlist is available in their documentation and from here.

After saving the configuration, Better Stack will begin scraping the endpoint. Your JuiceFS metrics should be received within a few seconds.

Creating a dashboard with AI SRE

With your JuiceFS metrics flowing into Better Stack, it's time to visualize them. You could build a dashboard manually, but Better Stack provides a smarter and more efficient way to do it by using AI SRE.

What is AI SRE?

AI SRE (Site Reliability Engineering) is Better Stack's chat-based site reliability assistant. It's an autonomous AI agent that can read your telemetry data, analyze incidents, build dashboards, and even write code to fix errors. Instead of waiting for humans to manually set up charts and queries, AI SRE can generate comprehensive dashboards for you based on a prompt.

It's notable that AI SRE is a paid feature. If you're on the free plan, you can still create dashboards manually using the drag-and-drop chart builder.

Creating a JuiceFS monitoring dashboard with a single prompt

Once your metrics source is ready, follow these steps:

From the left panel, head to Telemetry and then Metrics.
Click Create dashboard and select the Create with AI option.
In the prompt field, give AI SRE a clear description of what you need. For example: "Create me a dashboard to track ALL JuiceFS metrics, such as latency, data size, etc."
Also make sure to select the metrics Source you created earlier (for example, "jfs-better-stack") so that AI SRE has the proper context and data to work with.

Give the platform a few minutes for the dashboard to be created. AI SRE will analyze your JuiceFS metrics and automatically generate a complete set of charts and panels for the important performance indicators such as throughput, IOPS, latency, and storage utilization. For my first time trying this, it just worked like a charm as shown below.

AI SRE is a powerful feature that does so much more than create dashboards. It can analyze incidents, perform root cause analysis, suggest fixes, and even open pull requests. We've only scratched the surface in this post. This is your first step toward a smarter, AI-assisted observability workflow. After building your dashboard, you can further customize it by adding panels, editing queries, or setting alerts directly from the graphs.

Conclusion

In this post, we have walked through how to build a complete observability system for JuiceFS with Better Stack. We started by setting up the JuiceFS file system and getting its Prometheus-formatted metrics, then created a metrics source in Better Stack to ingest the data. We examined rapid creation of a full dashboard with AI SRE.

We hope this guide helps you gain better visibility into your JuiceFS deployment. If you have any questions or run into issues, we'd love to hear from you. Join the JuiceFS community on GitHub or Discord. And don't forget to check out Better Stack's documentation and their amazing YouTube channel for practical insights about distributed file storage, observability, AI, and more.

JuiceFS 1.4: Faster Metadata Operations with Batch Unlink, Batch Clone, and Redis Client-Side Caching

DASWU — Thu, 18 Jun 2026 08:30:56 +0000

In large-scale file access scenarios such as AI training and dataset management, metadata often becomes the first performance bottleneck as file counts and concurrency grow. Whether you're deleting millions of small files, cloning large datasets, or traversing directories under heavy concurrency, metadata performance directly impacts application efficiency.

JuiceFS Community Edition 1.4 introduces three major metadata optimizations:

Batch unlink for large-scale file deletion
Batch clone for metadata cloning
Redis client-side caching for hot metadata reads

These improvements reduce transaction commits, network round trips, and redundant metadata lookups. In tests on a flat directory containing 100,000 files, batch unlink improved performance by up to 93×, while batch clone achieved up to 24× speedup.

In this article, we’ll explain the motivation, design, and performance benefits behind these optimizations.

Deletion: From one‑by‑one to batched transactions

Under JuiceFS' metadata-data separation architecture, deleting a file involves much more than removing a directory entry. The system must also:

Update inode reference counts
Reclaim inode and space resources
Process trash entries
Update quota statistics

These operations must typically be completed within the same transaction.

When a directory contains hundreds of thousands or even millions of files, the traditional file-by-file deletion approach used by rm -rf quickly becomes a bottleneck. Each unlink request goes through the FUSE protocol, switches between kernel and user space, and triggers a separate metadata transaction.

As the number of files grows, the overhead from system calls, context switches, network round trips, and transaction commits accumulates rapidly.

To mitigate this issue, JuiceFS previously introduced the juicefs rmr command. Unlike rm -rf, rmr bypasses the FUSE layer and sends deletion requests directly to the client. It also supports multi-threaded deletion (50 threads by default), significantly improving throughput.

However, each file deletion still requires its own metadata transaction. Deleting 100,000 files still means executing 100,000 transactions.

Batch unlink takes optimization one step further by merging many independent deletion operations within the same directory into a single batch transaction, further removing network overhead.

Core design

The key is to turn many small transactions into fewer large ones. JuiceFS adds a batch unlink interface at the metadata engine layer. It allows the client to delete multiple non‑directory files under the same directory in one call.

When recursively clearing a directory, JuiceFS reduces deletion overhead in two ways:

Different subdirectories are handled concurrently with multi‑threaded deletion.
Inside each directory, normal files and symlinks are grouped into batches and sent to BatchUnlink.

This merges many unlink operations into fewer batch transactions at the metadata level.

It's important to note that BatchUnlink does not directly delete directories. Directory removal still follows the standard recursive workflow: empty the subdirectory first, and then delete the subdirectory itself. Therefore, BatchUnlink only applies to regular files and symbolic links within the same directory.

This restriction preserves correct recursive deletion semantics while avoiding consistency risks to the directory tree structure.

Implementation across metadata engines

JuiceFS uses different batching strategies depending on the metadata backend to minimize transaction commits and network round trips.

SQL backends (MySQL, PostgreSQL, etc.): Previously, each file deletion required its own sequence of INSERT, DELETE, and UPDATE statements. With BatchUnlink, the system:

Fetches all edge records for the target entries in a single batch query.
Retrieves the relevant inode attributes in a single locked batch query.
Executes edge deletions, inode state updates (decrementing nlink or marking for cleanup), and delfile entry insertions — all within one transaction.

Instead of executing one transaction per file, the entire batch can now be completed in a single transaction.

Redis backend: The optimization uses Redis pipelines and transactions. Where individual deletions previously required separate command round trips, BatchUnlink collects all HDEL (dentry removal), ZADD (enqueue for cleanup), SET (inode attribute update), and INCRBY (counter update) commands for multiple files into a single pipeline, executed atomically within one MULTI/EXEC transaction. To avoid blocking Redis' single-threaded event loop for too long, batch size is capped at 250 entries.

TiKV backend: BatchUnlink consolidates multiple deletions into a single transaction, using TiKV's batch write capability to reduce network round trips and transaction overhead. For distributed key-value backends, this kind of batching allows the backend's concurrent write capacity to be more fully utilized.

The figure below shows benchmark results on a flat directory of 100,000 files using juicefs rmr --threads 16. BatchUnlink delivers meaningful improvements across all metadata backends, with TiKV and Redis showing the largest gains.

Clone: From one‑by‑one copy to batched references

juicefs clone creates fast copies of files or directories for training dataset version management, experiment snapshots, and large-scale directory duplication. Its efficiency comes from the fact that cloning doesn't immediately copy the underlying data blocks. Instead, it creates new file records at the metadata layer and reuses the source file's existing block references. New data blocks are only allocated when the clone is actually written to. This avoids the time and storage overhead of a full copy.

For large directory clones, the same problem as deletion arises: processing files one by one generates a large number of short transactions and network round trips. The core idea behind batch clone is to merge the clone operations for multiple files in the same directory into a single batch transaction. When recursively cloning a directory, the system reads directory entries in batches as a stream. For each batch, all non-directory entries are collected and cloned together in one operation.

One key implementation detail is inode pre-allocation: before entering the transaction, the system uses nextInode to pre-allocate target inodes for all entries to be cloned. This avoids lock contention from repeatedly requesting inodes inside the transaction. Once inside the transaction, the system batch-queries all source file attributes (with row locks), builds all the insertion data for target nodes, edges, chunks, symlinks, and xattrs, and then inserts everything in a single batch.

Batch clone uses each backend's native batch write capabilities in a similar way to batch unlink. The per-backend implementation details won't be repeated here.

The performance gains vary across backends depending on:

Transaction models
Network communication overhead
Batch insertion efficiency for metadata records such as nodes, edges, and chunk references

Results on a flat directory of 100,000 files are shown below. MySQL sees the largest improvement at approximately 24x; Redis at approximately 5x; TiKV at approximately 2x.

Redis client-side caching: Keeping hot metadata local

In high-concurrency metadata workloads such as AI training dataset access and large-scale container startup, network round trips between JuiceFS clients and Redis often become a major performance bottleneck.

Consider the following operation:

open("/mnt/jfs/dataset/images/cat.jpg")

Before the file can be opened, the Linux Virtual File System (VFS) must resolve every component in the path:

Look up dataset.
Look up images.
Look up cat.jpg.

If the images directory contains hundreds of thousands of files and training jobs perform random access across the dataset, each lookup requires a GET request to Redis.

Under heavy concurrency, this results in large numbers of network round trips and increased Redis CPU utilization. Even though a single Redis query takes only a few dozen microseconds, network latency pushes each lookup to hundreds of microseconds or even milliseconds. When thousands of training processes are accessing files simultaneously, this overhead becomes significant.

How it works: Redis 6.0 client-side caching

Redis 6.0 introduced client-side caching, which allows clients to cache hot keys locally and receive invalidation notifications whenever those keys are modified.

Based on this capability, JuiceFS caches two categories of metadata in client memory:

Inode attribute cache. Keyed by inode number, this stores the complete attribute data for a file, such as type, size, permissions, and timestamps. The caching is implemented transparently through hook mechanisms in the Redis driver layer. On query, it first checks the local cache; on hit, it returns immediately without any network request. On modification, it automatically invalidates the corresponding cache. Application logic requires no awareness of the cache.
Directory entry cache. Keyed by "parent inode + path separator + filename," this caches the results of directory lookups. Unlike the inode attribute cache, the lookup logic for entry cache is embedded directly in the directory lookup path rather than being intercepted transparently at the driver layer. When entries for a directory are invalidated, all related cache entries under that directory are cleared using prefix matching. This allows path resolution and repeated access to hot entries in the same directory to be served from local memory.

Introducing client-side caching creates a consistency challenge in multi-mount scenarios. When multiple clients share the same JuiceFS file system, an operation on one client — creating, deleting, renaming, or updating attributes of a file or directory — can invalidate cached inode attributes or directory entries on other clients. Without an effective invalidation mechanism, subsequent reads could hit stale metadata, causing the directory entries or file attributes seen by one client to diverge from the actual state in the backend.

To address this, JuiceFS introduces a Tracking and Broadcast Invalidation (BCAST) model on top of Redis' client-side caching mechanism. After connecting to Redis, each client declares the metadata key prefixes it wants to track. When those keys are modified, Redis sends invalidation notifications to the relevant clients. On receiving a notification, the client clears the corresponding inode attribute cache or entry cache entries, so that subsequent accesses fetch fresh data from the metadata engine.

In addition, at client initialization, JuiceFS warms up metadata for the root directory of the mount point. Since these files are typically the most frequently accessed, benchmarks show this warm-up significantly improves overall access performance.

Through this mechanism, hot metadata can be reused locally. When the metadata changes, the related caches are evicted in time, reducing the risk of stale metadata.

When to use it

Redis client‑side caching works best in read‑heavy, write‑light scenarios with repeated access to hot metadata. AI training dataset loading is a good example: the dataset is usually read‑only during training, and tasks repeatedly access the same directories and files, so inode attribute cache and entry cache hit often, reducing redundant lookups and remote metadata queries.

The benefit is even more obvious when there is higher network latency between the client and the Redis metadata engine, such as in cross-availability-zone deployments.

Redis 6.0 or later is required to use this feature. The default cache expiration time is 1 minute, which provides a safety net in case of network interruptions or connection anomalies where invalidation notifications may not arrive, preventing stale entries from persisting indefinitely. For workloads with stricter consistency requirements, the expiration time can be shortened or client-side caching can be disabled entirely to reduce the risk of reading stale metadata.

Summary

These three optimizations each target a different path through the metadata layer:

Batch unlink merges multiple independent unlink operations within the same directory into a single batch transaction.
Batch clone merges multiple independent clone operations within the same directory into a single batch transaction.
Redis client-side caching keeps hot metadata in client memory, bringing read latency from network-level down to memory-level, with broadcast invalidation to maintain consistency across multiple clients.

BatchUnlink and BatchClone are internal interfaces. Users do not call them directly. Just use the right commands: juicefs rmr for deleting large directories, juicefs clone for copying directories. The optimization will be applied automatically.

One thing worth noting: both batch operations work by merging regular files within the same directory into a single batch transaction. Subdirectories are handled recursively by concurrent goroutines. The larger the directory, the greater the benefit.

Batch operations mainly merge ordinary files under the same directory into one batch transaction. Subdirectories are handled recursively by concurrent goroutines. The larger the directory, the bigger the benefit.

All optimizations above are available in JuiceFS Community Edition 1.4. Upgrade the client to get the performance gains.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

How Gongjiyun Keeps Model Distribution Fast Enough for Cross-Cloud Elastic Inference

DASWU — Fri, 12 Jun 2026 03:25:32 +0000

Founded in 2023 at Tsinghua University, Gongjiyun provides compute platforms and Model as a Service (MaaS) for artificial intelligence generated content (AIGC) enterprises and research institutions. We aim to alleviate the mismatch between elastic compute demand and supply. By aggregating idle IDC resources and edge resources, the platform offers containerized services, delivering rapidly schedulable compute for volatile workloads such as AI inference, video rendering, data processing, and data synthesis.

In cross-cloud elastic inference scenarios, compute tasks can be scheduled to different regions, cloud environments, and clusters, but model files and application data are large and cannot be migrated as quickly as compute resources. Especially in online inference, the model repository is read‑heavy and frequently accessed – storage access performance directly affects service startup, elastic scaling, and request latency.

To address this, we built an object storage acceleration solution on top of JuiceFS, integrating users’ existing object storage into elastic inference clusters. Through a unified namespace, metadata import, FUSE mount, distributed cache, and data warm-up, it improves access efficiency for model repositories across clouds and clusters. In a case study with a leading text‑to‑image model community, the solution supports a tens‑of‑TB model repository, dynamic loading of checkpoints and low-rank adaptations (LoRAs), and elastic scaling of hundreds of GPUs at peak, while keeping additional latency within the customer’s acceptance range.

In this post, we'll walk through why storage — not compute — is the real bottleneck in cross-cloud elastic inference, how we evaluated and chose JuiceFS, and the step-by-step optimizations that brought latency from +10s down to under 2s in production.

Elastic demand is widespread, but supply is hard to match

As AI applications grow rapidly, compute demand continues to increase, but resource usage patterns differ across scenarios. Compared to training, which has stable resource needs, AI inference, data processing, and data synthesis are often more volatile: office applications may see higher traffic during the day, entertainment apps during evenings or weekends, and project‑based data processing may consume large amounts of compute in short bursts then idle. For small teams or exploratory applications, elastic compute also helps them better evaluate the relationship between per‑request cost and application value.

On the supply side, compute infrastructure is capital‑intensive. Resource providers are not incapable of offering elastic services, but they prefer long‑term dedicated leases to recover costs and reduce risk. As a result, low price, stability, and elasticity are difficult to achieve together:

Dedicated leases are low‑cost and stable but lack elasticity.
Spot resources are cheap and elastic but uncertain.
On‑demand resources are elastic and stable but expensive.

In China, this contradiction is further reflected by a market dominated by dedicated leases, with elastic supply accounting for a small share.

We aim to resolve this mismatch between elastic demand and supply. By aggregating idle IDC and edge resources, the platform offers containerized services, providing rapidly schedulable compute for AI inference, video rendering, data processing, and data synthesis. At lower resource costs, we help users quickly spin up tasks during peaks, schedule them across clusters, and handle elastic demand, while enabling resource providers to improve utilization and monetize idle capacity beyond dedicated leases.

Compute can be scheduled: How does storage keep up?

As elastic compute platforms evolve, compute resource scheduling is easy. Container images can be synchronized across clusters via registries and distribution networks, tasks can be launched in different resource pools by schedulers, and traffic can be distributed via unified ingress and traffic management.

But model and data files are typically large, making cross‑cloud, cross‑cluster migration costly and slow, unable to match the sub‑second startup and release of compute. Therefore, in cross‑cloud elastic inference architectures, the real limitation on system elasticity is often not compute scheduling, but the efficiency of data and model distribution.

Different application scenarios have different storage requirements:

Model training, development, and debugging: These involve complex read‑write needs, including code repositories, model files, experiment results, and intermediate state. They also require high environment stability; users cannot tolerate state loss from frequent host switching. Thus, the platform typically provides long‑term stable compute resources and runtime environments, and storage needs can be met by existing stable storage systems.
Data processing: This can be split further. If a single processing job has high application value and can cover cross‑cloud network transfer costs, you can build a pipeline that continuously pulls data from S3 or other object storage, processes it in the compute cluster, and writes back streaming. The system does not need large local storage. If the data scale is larger or per‑job value is low, local storage acts as a one‑time cache. Data flows through and does not need to be persisted.

What is truly more challenging is the online inference scenario. Online inference services cannot tolerate downtime. However, the resources used by an elastic computing platform may come from idle resource pools. These resources could be preempted. Once resources in a certain data center or cluster become unavailable, the platform must be able to migrate tasks to other providers or other clusters in time. This means not only computing tasks must be migrated. Model files and related storage access capabilities must also be migrated at the same time

Online inference has higher requirements for service continuity and cross-cluster migration capabilities, but its storage access pattern is also more clear. Compared to training, development, and debugging scenarios, inference workloads are typically read heavy. The core needs focus on efficient model loading, reading model weights, and accessing the model repository. For large models and online applications, model loading speed directly affects service startup time, elastic scaling efficiency, and request response stability. Therefore, inference scenarios are not suitable for simply adopting traditional read-write hybrid storage architectures. Instead, they are better suited for specialized optimizations around model distribution, read only access, and cache acceleration.

In addition, an elastic computing platform usually does not host a user's complete application system. The user's primary cloud account, application database, model management system, and even some fixed computing resources often already exist in other clouds or on premises. For the platform to integrate with the user's application, it must be compatible with the user's existing model repository and model management processes. It cannot require the user to fully migrate the entire system.

Therefore, to support cross-cloud elastic inference, we need more than just compute scheduling capabilities. We need a cross-cloud high-performance storage and model distribution solution tailored for model inference scenarios. This solution must support hosting a large model repository and high-performance reading, it must adapt to the user's existing model management system. And it must provide stable data access capabilities when resources are migrated across clouds and clusters.

Why JuiceFS: Unified cross-cloud access, strongly consistent metadata, and high-performance cache

Facing cross-cloud elastic inference scenarios, the storage system needs to meet several conditions at the same time:

It must provide a unified access point across different clouds and clusters. It must support shared read-write access and unified metadata management.
It must be compatible with the user's existing object storage and model repository to avoid data migration.
It needs low operational complexity and good read performance.

When evaluating storage options, we considered Ceph:

Ceph is mature. It’s suitable for building unified storage within a single data center or a stable resource domain.
However, in cross cloud elastic inference scenarios, Ceph requires high network stability and operational skills. The overall integration cost is higher. So we did not choose it.

We also evaluated Alluxio. However, in a multi-cloud environment, multiple clusters need to access the same underlying object storage data concurrently. The workload is not purely read only; there are also occasional writes. This scenario requires strong data consistency. Therefore, Alluxio was not chosen for production.

We finally chose JuiceFS mainly because:

It uses object storage as the database.
It provides a unified namespace and consistent file system view through an independent metadata service. This allows multiple clusters to access the same model data as a file system.
This architecture is suitable for cross-cloud and cross-cluster model distribution and shared reading.
It’s also compatible with the user's existing object storage and model repository, reducing data migration and application integration costs.

The decision to further adopt JuiceFS Enterprise Edition was mainly due to its distributed caching capabilities and managed metadata service. In this scenario, the value of JuiceFS is not just providing a file system interface. It combines object storage, unified namespace, metadata management, and cache acceleration into a storage access layer that is better suited for cross-cloud elastic inference.

Practical: Object storage acceleration based on JuiceFS

Based on JuiceFS, the platform encapsulates an object storage acceleration product. This product connects the user's existing object storage to the elastic inference cluster. It provides the storage as a high-performance file system for the application. The overall process is as follows.

Create a file system. The user provides object storage access credentials, for example, AK/SK for S3-compatible storage. The credential permissions can be configured as read only or read-write based on application needs. The platform creates a corresponding JuiceFS file system based on that object storage.
Import metadata. The platform uses the JuiceFS import feature to scan the metadata of files in object storage. Then, it imports that metadata into the JuiceFS metadata service. In this way, the model files originally stored by the user in object storage can be accessed as file system directories in JuiceFS.
Create a cache group. Within each cluster that may host workloads, the platform sets up a JuiceFS cache group. This forms a distributed cache group. Before running a task, the platform can warm-up model files. It caches hot data in the target cluster in advance. This reduces the time needed to pull data from remote object storage when the inference service starts.
Mount to application Pods. When the user's application runs, the platform uses the FUSE client to mount the JuiceFS file system into the application Pod. For the application, model files appear as local file system paths. Therefore, the original model reading logic usually does not need modification.
Enable node local cache. Besides the cluster level cache group, the node where the FUSE client runs can also provide local cache. This improves repeated read and model loading performance. It further reduces direct access to remote object storage.

This object storage acceleration product essentially productizes the JuiceFS metadata import, distributed cache, data warm-up, and FUSE mounting process. It allows the user's existing object storage to serve cross-cloud inference tasks in a way that feels closer to a local file system.

In addition, the JuiceFS cache group is independent from the file system access point. This characteristic, on one hand, adds management complexity on the platform side, because the platform needs to manage the relationships among the file system, cache groups, mount points, and task scheduling. On the other hand, it provides a foundation for cache isolation, independent scheduling, and fine-grained management based on clusters, users, or application scenarios in the future.

Production case study: A leading text-to-image model community

Scenario, challenges, and acceptance criteria

One of the most representative cases in this object storage acceleration solution involves a leading Chinese text-to-image model community hosting tens of terabytes of model data, including large checkpoint base models and a larger number of smaller LoRA models. In practice, inference jobs typically load a checkpoint first, then load one or more LoRA models to perform combined inference.

The company already operated compute infrastructure at scale — several thousand GPUs — but its workload, serving creative design and production use cases, exhibited significant variability. Overall average utilization was below 50%, yet during morning and afternoon peak hours on weekdays, load could reach 140% of normal capacity, degrading the user experience. The customer therefore needed a highly elastic compute supply.

We provided a high-elasticity resource model: compute support at the scale of hundreds of GPUs was available only during weekday peak hours — 10:00–12:00 AM and 2:00–6:00 PM — with resources scaling to zero at all other times.

This meant the platform needed to provision hundreds of GPUs within a window of minutes, while consuming zero resources outside peak hours. For the customer, this model delivers large-scale compute during peak periods while avoiding payment for idle capacity. For the platform, it enables more efficient utilization and monetization of idle compute resources.

The technical challenges were significant:

A model repository of this scale cannot simply be replicated to every elastic cluster.
Inference services do not load all models once at startup. Model reads and switches happen continuously as user requests arrive, resulting in high access frequency. Therefore, the object storage acceleration solution needed to support not just large-scale model repository access, but stable read performance under continuous dynamic loading.

The customer's performance requirements were also strict. During acceptance testing, a portion of production traffic was routed to the elastic cluster. The requirement was that both the median and mean inference latency of the elastic cluster must stay within 2 seconds of the customer's own cluster. Given that individual inference jobs take on the order of tens of seconds, this requirement left virtually no room for additional latency introduced by the storage layer. In the first few rounds of testing, both median and mean inference latency on the elastic cluster exceeded the customer's own cluster by approximately 10 seconds — failing the acceptance criteria.

Performance optimization: Reducing additional latency on the elastic cluster

Optimization began with the median. A high median indicates that a significant proportion of requests are experiencing performance degradation, not just a small number of outliers inflating the tail. JuiceFS monitoring revealed that the cluster's cache hit rate was not reaching the expected level. In the current architecture, a cache miss requires a round trip over the public internet to the customer's object storage on Alibaba Cloud. This significantly increases model loading time and then affects inference request latency.

To solve this, the platform used the isolation capability of the JuiceFS cache group. It assigned dedicated cache nodes to this customer, reserved enough cache space, and warmed up the core model data. After warming up, the access path for core models achieved nearly 100% cache hit rate. This effectively avoided the performance loss from cross public network backfilling.

The second factor affecting the median was metadata access latency. Because the platform uses a unified cross-cluster architecture, the metadata service is accessed over the public internet, for example, via JuiceFS Cloud Service or a deployment on a remote host, and this latency affects overall model read performance.

The platform took two measures to address this issue:

Enabling JuiceFS' open cache to keep metadata in local memory as much as possible. Since this workload is predominantly read-only, caching is an effective way to reduce metadata access overhead.
Tuning the cluster's network rate-limiting policy. While the platform cannot directly control network equipment in edge data centers, it can apply node-level rate limiting to prevent any single node from saturating the available bandwidth, improving overall network stability. After these optimizations, cluster-wide performance improved meaningfully and the median metric gradually reached the customer's requirement.

Once the median met the target, the mean still showed a gap. This indicated that long-tail requests remained, with a small number of requests taking significantly longer than normal and pulling up the overall average. Further analysis traced this to node-level local cache — specifically, the FUSE cache quota. With limited cache capacity, the elastic cluster experienced more frequent cache evictions than the customer's own cluster, causing some requests to reload model data from scratch and increasing mean inference latency. The platform addressed this by increasing the FUSE local cache quota in the production environment, reducing eviction frequency, improving tail latency, and ultimately bringing the mean metric within acceptance. The system passed validation and has been running stably since.

Multi-tenant cache management

After the single-tenant case was validated, the solution entered multi-tenant operation. As different tenants began time-sharing the same elastic nodes, a new issue emerged: cache contention between tenants.

In the elastic resource model, FUSE clients do not actively clear node cache on exit. This is a reasonable design in single-tenant scenarios, where cached data from previous jobs can be reused by subsequent jobs to improve hit rates. However, in multi-tenant scenarios, one tenant's data can occupy node cache for extended periods. This leaves insufficient cache capacity for the next tenant, who is then forced to fall back to object storage, causing a noticeable performance drop.

To address this, we deployed an independent daemon process on each node that performs a global cache garbage collection (GC) pass before the application FUSE client starts. The eviction strategy references the JuiceFS FUSE client implementation, using a 2-random policy to balance collection efficiency and performance overhead. Coordination across nodes is handled via Kubernetes distributed locks: only the client that acquires the lock executes GC, preventing multiple clients from running cache collection simultaneously and creating excessive network and I/O pressure.

This mechanism effectively mitigates the problem of historical jobs occupying cache resources in multi-tenant scenarios, allowing different tenants sharing elastic resources to maintain consistent cache performance.

Conclusion

For elastic compute to reliably serve production traffic, compute scheduling alone is not enough. Model data and hot data must remain stably accessible across clouds and clusters.

Built on JuiceFS, we’ve combined object storage, unified namespace, metadata management, distributed caching, and FUSE mounting into an object storage acceleration solution purpose-built for elastic inference. This is not simply about mounting object storage as a file system. It’s about building a data access layer around the access patterns of model inference: one that supports warm-up, caching, isolation, and management.

This represents Gongjiyun's current progress in elastic compute and cross-cloud storage acceleration. As AI inference scenarios continue to evolve, model distribution, cache management, and multi-cluster data access will continue to surface new engineering challenges. We look forward to exchanging ideas with developers, AI application teams, and infrastructure practitioners, and to exploring more stable and efficient data access solutions for elastic compute environments.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

DEV Community: DASWU

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

Supporting 1 million clients

Fast cloning for large directories

Performance improvements

More than 90% RDMA bandwidth utilization

Higher random-read IOPS under high concurrency

Optimizing data access across regions

On-demand synchronization to reduce data replication

Accelerating metadata access with read-only nodes

More flexible cache management

Improving hot-data cache hit rates

Preserving cache hit rates when adding multiple cache nodes

Reducing the number of files on large cache disks

Warming up specific byte ranges

Simplifying operations for massive files

Restoring files by deletion time

Reducing GC and fsck memory usage

Restricting access with token allowlists

Summary: scale changes everything

Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

The creative trio: JuiceFS + SQLite + Litestream

An unconventional but effective combo

The performance reality

Final thoughts

Keeping GPUs Fed: How Meta's AI Storage Architecture Aligns with JuiceFS

Introduction

Why storage matters for AI workloads

Meta's AI storage blueprint: the evolved architecture

The three-component architecture

Cross-region data access with tiered caching

JuiceFS: cloud-native, AI-native from day one

Cross-region strategy: mirrors and replication

Architectural similarities

A concrete example: explicit cache hydration

Conclusion

Giving AI Agents a Real File System

The new root user

Architectural paradox

Why AI agents love the file system

Offloading the context window

The universal abstraction

Unlocking the Unix ecosystem

Database-driven metadata meets elastic storage

The ecosystem landscape

Lightweight embedded approach: AgentFS with SQLite

Client-server transactional approach: TigerFS with PostgreSQL

Decoupled distributed approach: JuiceFS with object storage

Agents' full toolbox

Final thoughts

GPFS vs. Alluxio vs. JuiceFS: Architecture and Use Cases Compared

I/O patterns and storage challenges in AI workloads

Autonomous driving: large‑scale data generation and training

LLM training: full‑pipeline data access

Multimodal models: coexistence of small files, aggregated data, and model files

Compute platforms: multi‑cloud collaboration and model distribution

Quantitative finance: balancing performance and cost

AI agents: data sharing in short‑lived sandboxes

GPFS vs. JuiceFS

From PFS to GPFS: capabilities and limitations of parallel file systems

Architecture advantages and trade-offs: metanode, token locks, and strong consistency

Performance comparison

Sequential reads: GPFS better per node, JuiceFS scales through cache groups

Sequential writes: GPFS excels at synchronous writes; JuiceFS uses writeback for scale

Random reads: GPFS better at high concurrency, JuiceFS competitive in certain scenarios

Random writes: GPFS superior at high concurrency

Selection summary

Alluxio vs. JuiceFS

Key architectural differences

Data organization and consistency boundaries

Cache and namespace organization

Metadata architecture

How architectural differences affect real-world usage

POSIX compatibility

Write performance and write amplification

Deployment and engineering capabilities

Selection recommendations

Summary

5.5 Faster Small-File Writes: Tuhu Built a Unified AI Storage Platform with JuiceFS + Ceph RADOS

From multiple storage systems to a unified storage foundation

Reducing GC and `fsck` memory usage