DEV Community: sy z

Why did we develop HPFS?

sy z — Fri, 21 Jun 2024 03:24:35 +0000

Why did we develop HPFS? When training with ResNet-50, we use a large number of images, reaching a scale of hundreds of millions. We also need to train the Stable Diffusion model, which requires a staggering amount of data, ranging from tens of billions to hundreds of billions of datasets. We have tried using Lustre, Ceph, GlusterFS, and IBM's GPFS, but all have failed because, at this scale, the performance of all file systems degrades severely, to the point of being unusable. Moreover, our concurrency is very high, with up to hundreds of threads running simultaneously. This is determined by the nature of model training, where a batch of data is simultaneously given to several training models to run concurrently. You can think of it as multiple clients running highly concurrently. In this context, we developed HPFS to meet our high-load training tasks. We do not use the FUSE version of HPFS because of performance limitations; instead, we modified the training model's interface to call the HPFS API for training. In summary, the performance difference between the API interface and the FUSE client is significant.

Today, I will share the performance of these open-source file systems in our use. CephFS with multi-MDS does not scale linearly and requires enough memory to maintain performance, which is quite awkward. Lustre is limited by the central node's performance bottleneck. GlusterFS is limited by the performance of the local file system it relies on, such as EXT4 or XFS, which degrades significantly with many files. GPFS's performance is stable from start to finish but very slow. In the future, we may open source HPFS to help solve the storage problems of massive datasets.

Our performance in use is approximately as follows:

Eight clients, each running 128 threads. HPFS-SRV uses three machines, each running 16 HPFS-SRV instances. The storage data is a cluster of six machines with 8 NVMe each.
Test case: open, write 4096 bytes, close, then open, read 4096 bytes, close.

Lustre: 9k
CephFS (multi-MDS): 25k
GlusterFS: 16k
GPFS: 10k
HPFS: 1700k
hpfs github url:
https://github.com/ptozys2/hpfs

hpfs vs cephfs performance

sy z — Tue, 18 Jun 2024 05:01:46 +0000

In recent days, I conducted a performance test of HPFS. Under the same environment, I also deployed CephFS and made a performance comparison. The test case was to open a file, write 4096 bytes, close it, then open it again, read 4096 bytes, and close it. This continuous operation was repeated to create a total of 100 million files of 4096K each, with multiple threads operating concurrently. The IOPS was measured from this test.

From this test data, we can see that HPFS performance increases linearly with the addition of clients. This is because the capability of a single client is limited by the bottleneck of FUSE. If HPFS's API interface is used, this limitation is completely removed, and the performance of all NVMe disks can be fully utilized.

On the other hand, CephFS is limited by the bottleneck of MDS, and its IOPS cannot increase even with the addition of more clients. HPFS, however, can continue to scale its metadata file system's IOPS load capacity by expanding the number of HPFS-SRV.

github url:
https://github.com/ptozys2/hpfs

For an introduction to HPFS, please refer to this article：
https://dev.to/sy_z_5d0937c795107dd92526/multi-meta-server-ceph-4pe1

multi meta server ceph

sy z — Fri, 14 Jun 2024 10:02:03 +0000

HPFS is a file metadata server based on Ceph RADOS, designed to address the bottleneck issues commonly encountered in Ceph file systems. Unlike traditional setups that rely on third-party storage for metadata, HPFS directly stores metadata on RADOS. This approach ensures metadata stability akin to that of Ceph, which has undergone over a decade of iterative development and is known for its robust data stability. By avoiding third-party plugins for metadata storage, we eliminate potential security risks associated with external data handling. One of the known shortcomings of Ceph is the single-point issue with its Metadata Server (MDS), which tends to underperform in high IOPS scenarios. HPFS has redesigned the metadata architecture to create a distributed framework, allowing metadata access to scale with the concurrent access capabilities of RADOS objects. Additionally, HPFS has been specifically optimized for scenarios involving intensive small file operations, such as AI visual training and graphic rendering, significantly enhancing IOPS performance under multi-client and concurrent conditions—often outperforming CephFS by orders of magnitude. If you manage environments with hundreds of millions of files requiring frequent metadata access, consider evaluating the performance of HPFS. We have now made some of HPFS’s features available for free to facilitate community interaction and collaborative discussion.

github url:
https://github.com/ptozys2/hpfs