Turn a Directory into an S3 Service with One Script: FluxonFS Delivers 5 Alluxio's Throughput

Project URL: https://github.com/Tele-AI/Fluxon

From loading models in AI training frameworks and archiving logs to lightweight backups on a personal NAS, S3 has become the de facto object storage API for modern cloud-native and AI tooling. For years, MinIO was the default choice for self-hosted S3: it was fast, broadly compatible, and easy to deploy. More recently, however, MinIO moved from the permissive Apache 2.0 license to AGPLv3, removed the management console from its open-source edition, and tightened its commercial offering. For many small teams and independent developers, it no longer feels like the obvious default it once was.

RustFS emerged to fill that gap. It reimplements the core object storage engine in Rust, providing compile-time memory safety without garbage-collection pauses. Its strong performance under concurrency, low latency, operational reliability, and active community have quickly made it a credible alternative to MinIO.

Yet MinIO and RustFS are both standalone object stores. Once data is written, the service takes ownership of it and stores it in an internal format and sharded directory layout. That model is well suited to large enterprise clusters, but it can feel disconnected for personal use, lightweight development, or local AI debugging: users cannot meaningfully inspect file placement with ls, search logs with grep, or migrate the data directly with rsync.

Disclosure: I work on Fluxon at TeleAI. FluxonFS addresses this specific gap. It caches and accelerates access to files and objects while keeping the existing local directory authoritative, and layers an S3-compatible gateway on top.

Like RustFS, FluxonFS keeps its performance-sensitive work in Rust. Python handles Quick Start configuration and process orchestration; S3 request handling, file and object caching, and transport remain in the Rust core. The asynchronous gateway uses Axum, Tokio, and bytes. Eclipse iceoryx2 handles same-host IPC. Fluxon's fork of Moka provides cache control with dynamic-capacity support. For cross-node transfers, the RDMA path is built on pplx-garden, another Rust project, with automatic TCP fallback.

This keeps CPU-intensive request handling outside the GIL, avoids garbage-collection pauses on the hot path, and makes buffer ownership and cleanup explicit. It does not make every request zero-copy, and Rust alone is not a performance guarantee; the benchmark below measures the complete path.

The FluxonFS Model: The Local Filesystem Is the Source of Truth

FluxonFS provides a standard S3 API without taking ownership of the underlying data, so the same files remain directly accessible from the local filesystem. Alluxio was an early proponent of this broader model: a common access layer and intelligent caching can decouple applications from the underlying storage.

Unlike a standalone object store, FluxonFS sits directly on top of an existing local directory and requires no migration to a new storage format. The local filesystem remains the single source of truth. FluxonFS adds multi-protocol access and cache acceleration for files and objects without taking over data management. This article focuses on its S3-compatible gateway.

FluxonFS also reuses Fluxon's KV cache, shared memory, and RDMA data paths to accelerate hot-object access and cross-node transfers without changing data ownership.

Because FluxonFS works directly with the existing directory, it does not wrap, hide, or replace the underlying files:

• Protocol translation: FluxonFS handles S3 semantics such as SigV4 authentication and multipart uploads, translating S3 requests into POSIX reads and writes on local files.
• Access control: S3-based multi-tenant authentication and authorization add a consistent service-side security boundary around the local directory.
• Cache acceleration: The built-in KV cache accelerates access to hot data. The cache is never authoritative and can be discarded or rebuilt at any time.
• High-performance data paths: FluxonFS reuses Fluxon's RDMA and shared-memory capabilities. Shared memory reduces copies within a node, while RDMA can lower transfer overhead between nodes.

The same data remains accessible in two ways:

• To an S3 client: The directory appears as a standard s3:// bucket that supports object uploads, downloads, and listings.
• To a local user: The data remains a collection of ordinary files that can be accessed directly with Linux tools such as ls, grep, and rsync.

No data migration is required, and changes remain visible from both sides. This model fits modern AI workflows as well as lightweight operational environments.

FluxonFS S3 Read and Write Performance

This single-machine benchmark compares FluxonFS S3 with the Alluxio S3 Proxy using rclone v1.60.1. Each case ran three times; all 162 tests passed content, disk-I/O, and interference checks.

• Workload: 2,000 × 4 KiB, 256 × 1 MiB, and 32 × 256 MiB files—8 GiB total—at concurrency 1, 8, and 32.
• Durable PUT: Timing included the upload and syncfs on the destination NVMe filesystem, not just the HTTP response.
• Cold read: Each group restarted the service and evicted file pages with POSIX_FADV_DONTNEED; FluxonFS disabled asynchronous backfill and Alluxio used NO_CACHE.
• Hot read: Each 64 GiB application cache held about 8.26 GiB of test data. After an untimed warm-up pass, only the NVMe file pages were evicted. Timed runs required zero backend NVMe reads.

Bars show three-run means. The 4 KiB results use objects/s; 1 MiB and 256 MiB results use MiB/s. Each file-size panel has its own y-axis.

Benchmark Results

1. FluxonFS was faster than Alluxio in all 18 durable PUT and cold-read combinations. The advantage ranged from 36% to 726% for durable PUT and from 7% to 661% for cold reads.
2. Small and medium-sized files saw the largest gains under concurrency. For 4 KiB cold reads, FluxonFS reached 605, 3,539, and 4,259 objects/s at concurrency levels 1, 8, and 32. At concurrency 32, the 1 MiB cold-read workload reached 1,145 MiB/s, about 7.6× the Alluxio result.
3. Large-file cold reads ran closer to the NVMe limit. FluxonFS led by about 7%, 19%, and 42% at concurrency 1, 8, and 32.
4. Hot-read results were closest for medium and large files. FluxonFS led by 8–43% for 4 KiB objects; 1 MiB results were within 1% except for a 19% lead at concurrency 8, and 256 MiB results stayed within 1.2%.

How Directories, Objects, and Caches Map to One Another

FluxonFS implements this model through an export. The export_name becomes the bucket name, and each path relative to the export root becomes an object key.

FluxonFS Concept	S3 Concept	Local Filesystem	Two-Way Behavior
export_name	Bucket	Public name of the exported directory	A predefined export is exposed as a bucket
remote_root_dir_abs	Bucket root	Absolute path containing the real files	Remains authoritative; all reads and writes ultimately operate here
relpath	Object key	Path relative to the export root	S3 writes create real files, and S3 can read files created locally
username / password	Access Key ID / Secret Access Key	FluxonFS user credentials	Used for standard SigV4 authentication

By default, serve_s3_single_node uses the following mapping:

export_name = quick-start-export
remote_root_dir_abs = /data

s3://quick-start-export/llama/model.safetensors
                                 ⇅
/data/llama/model.safetensors

quick-start-export is a fixed bucket name; it is not derived from the directory name. Existing data does not need to be imported. Local tools and S3 clients operate on the same files directly.

Start FluxonFS with a Linux Script or Docker

Both options below expose a chosen directory as the quick-start-export bucket at http://127.0.0.1:26180/fs_s3.

Prerequisite: Start etcd and GreptimeDB

Both options use etcd for control-plane metadata and GreptimeDB for monitoring data:

docker volume create fluxon-s3-etcd
docker volume create fluxon-s3-greptime
docker run -d --name fluxon-s3-etcd --restart unless-stopped -p 22379:2379 -v fluxon-s3-etcd:/etcd-data quay.io/coreos/etcd:v3.5.0 /usr/local/bin/etcd --data-dir /etcd-data --name etcd0 --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://0.0.0.0:2379 --listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://0.0.0.0:2380 --initial-cluster etcd0=http://0.0.0.0:2380 --initial-cluster-state new
docker run -d --name fluxon-s3-greptime --restart unless-stopped -p 24000:4000 -v fluxon-s3-greptime:/greptimedb greptime/greptimedb:v0.15.1 standalone start --data-home /greptimedb --http-addr 0.0.0.0:4000 --rpc-bind-addr 127.0.0.1:4001 --mysql-addr 127.0.0.1:4002 --postgres-addr 127.0.0.1:4003

These ports have no authentication; restrict them to the local machine or a trusted network.

Option A: Linux with pip

On x86_64 Linux with Python 3.10 or later:

python3 -m pip install --upgrade fluxon-py

Create serve_s3_single_node.py:

from fluxon_py.quick_start import serve_s3_single_node

kv_master_config = {
    "etcd_endpoints": ["127.0.0.1:22379"],
    "cluster_name": "fluxon_s3",
    "instance_key": "fluxon_s3_master",
    "network": {
        "tcp_reactor_mode": "event_driven",
    },
    "port": 25100,
    "log_dir": "/path/to/state/kv-master/log",
    "monitoring": {
        "prometheus_base_url": "http://127.0.0.1:24000/v1/prometheus",
        "prom_remote_write_url": ["http://127.0.0.1:24000/v1/prometheus/write"],
        "otlp_log_api": {
            "otlp_endpoint": "http://127.0.0.1:24000/v1/otlp/v1/logs",
            "db_name": "public",
            "table_name": "fluxon_logs",
        },
    },
}

kv_owner_config = {
    "instance_key": "fluxon_s3_owner",
    "network": {
        "tcp_reactor_mode": "event_driven",
    },
    "contribute_to_cluster_pool_size": {"dram": 1073741824, "vram": {}},
    "fluxonkv_spec": {
        "etcd_addresses": ["127.0.0.1:22379"],
        "cluster_name": "fluxon_s3",
        "share_mem_path": "/dev/shm/fluxon-s3",
        "sub_cluster": "default",
        "large_file_paths": ["/path/to/state/kv-owner/large"],
    },
}

serve_s3_single_node(
    "/path/to/data",  # Local data directory exposed through S3
    "/path/to/state",  # Persistent Fluxon state directory
    kv_master_config=kv_master_config,
    kv_owner_config=kv_owner_config,
    export_name="quick-start-export",
    start_middleware=False,
    greptime_base_url="http://127.0.0.1:24000",
)

Run it with python3 serve_s3_single_node.py. The service stays in the foreground; press Ctrl-C to stop it.

Option B: Docker Desktop on Windows

Run this in PowerShell. The initial admin / admin credentials are only for first-time setup; do not expose port 26180 before changing them.

docker run -d --name fluxon-s3 `
  --restart unless-stopped `
  -p 26180:26180 `
  --shm-size 2g `
  --add-host=host.docker.internal:host-gateway `
  --mount "type=bind,src=C:\fluxon-s3\data,dst=/data" `
  --mount "type=bind,src=C:\fluxon-s3\state,dst=/state" `
  --entrypoint python3 `
  "hanbaoaaa/fluxon_quick_start:0.2.4" `
  -c "
from fluxon_py.quick_start import serve_s3_single_node

kv_master_config = {
    'etcd_endpoints': ['host.docker.internal:22379'],
    'cluster_name': 'fluxon_s3',
    'instance_key': 'fluxon_s3_master',
    'network': {
        'tcp_reactor_mode': 'event_driven',
    },
    'port': 25100,
    'log_dir': '/state/kv-master/log',
    'monitoring': {
        'prometheus_base_url': 'http://host.docker.internal:24000/v1/prometheus',
        'prom_remote_write_url': ['http://host.docker.internal:24000/v1/prometheus/write'],
        'otlp_log_api': {
            'otlp_endpoint': 'http://host.docker.internal:24000/v1/otlp/v1/logs',
            'db_name': 'public',
            'table_name': 'fluxon_logs',
        },
    },
}

kv_owner_config = {
    'instance_key': 'fluxon_s3_owner',
    'network': {
        'tcp_reactor_mode': 'event_driven',
    },
    'contribute_to_cluster_pool_size': {'dram': 1073741824, 'vram': {}},
    'fluxonkv_spec': {
        'etcd_addresses': ['host.docker.internal:22379'],
        'cluster_name': 'fluxon_s3',
        'share_mem_path': '/dev/shm/fluxon-s3',
        'sub_cluster': 'default',
        'large_file_paths': ['/state/kv-owner/large'],
    },
}

serve_s3_single_node(
    '/data',
    '/state',
    kv_master_config=kv_master_config,
    kv_owner_config=kv_owner_config,
    export_name='quick-start-export',
    start_middleware=False,
    greptime_base_url='http://host.docker.internal:24000',
)
"

Replace both C:\fluxon-s3\... paths with existing Windows directories. Keep both bind mounts across restarts; /data is authoritative, while /state contains persistent Fluxon state. The shared-memory area is rebuildable.

After startup, the service exposes:

S3 endpoint: http://127.0.0.1:26180/fs_s3
Web UI:      http://127.0.0.1:26180/fs_s3/ui/
bucket:      quick-start-export

Log in to the Web UI with admin / admin and change the credentials. The new username becomes the S3 Access Key ID, and the new password becomes the Secret Access Key. Wait about two seconds for the new authorization state to reach the FS agent before connecting an S3 client.

Verify the Two-Way Mapping Between the Local Directory and S3

Point rclone at the printed endpoint, enable path-style URLs, and use the updated credentials. Three checks demonstrate that both interfaces operate on the same authoritative data:

1. Upload upload/from-client.txt to quick-start-export; the file appears immediately under the exported local directory.
2. Append text to that local file; the next S3 GET returns the new content.
3. Delete the object through S3; the local file disappears.

Current Compatibility and Operational Considerations

FluxonFS combines an S3 endpoint with a local-file workflow; it does not implement every AWS S3 enterprise feature. End-to-end tests with rclone v1.60.1 covered bucket checks, listings, uploads, downloads, deletion, and multipart uploads.

Compatibility summary:

• Supported: ListBuckets, HeadBucket, core ListObjectsV2 prefix/delimiter behavior, GET, HEAD, single-range requests, PUT, DELETE, multipart uploads, and header-based SigV4. PUT creates parent directories when needed; multipart parts are removed after completion or cancellation.
• Limited or unsupported: Clients cannot create buckets dynamically. Export names must be 3–63 characters long, use only lowercase letters, digits, or hyphens, and not begin or end with a hyphen. Full continuation-token pagination is not yet supported, and query-string presigned URLs are not guaranteed. Versioning, ACLs, tags, lifecycle policies, server-side encryption, and complete CopyObject semantics are unsupported. ETags are not necessarily MD5 digests.

These tests cover common read and write paths; they are not full AWS S3 compatibility certification.

Key considerations:

1. Durability depends on the underlying storage: Adding an S3 endpoint does not automatically add replication or high availability.
2. Concurrent writes require application-level coordination: Applications must handle atomicity and conflicts when S3 and local processes write the same file.
3. Public deployments require hardening: Change the initial credentials before exposure, restrict internal ports, and add TLS, monitoring, and backups.

Conclusion

FluxonFS is a Rust-based system that caches and accelerates access to files and objects while leaving existing directories in their native layout. Its S3-compatible gateway exposes those files to S3 clients without moving the data into a proprietary layout.

The wider Fluxon project reuses the same caching, shared-memory, RDMA, observability, and deployment infrastructure across KV/RPC, messaging, and file and object access.

Fluxon is licensed under Apache 2.0. Source code, benchmark scripts, and full configuration examples are available on GitHub.