DEV Community

Cover image for Object Storage vs File Storage: When to Use Which (2026)
Ethan Carter
Ethan Carter

Posted on

Object Storage vs File Storage: When to Use Which (2026)

#objectstorage #filestorage #s3 #nfs

I still see engineers storing user-uploaded photos in /var/www/uploads/ on an ext4 volume and wondering why their server falls over at 10M files. Meanwhile, the team next door threw the same photos into an S3 bucket and scaled to 100M files without breaking a sweat. The difference is the storage paradigm. Pick the wrong one and you feel it at scale.

I still mix the two up in conversation sometimes, so I keep a short checklist: random writes and file locks mean file storage; HTTP PUTs and billions of objects mean object storage.

Short answer: use file storage when you need POSIX semantics — in-place edits, sub-millisecond random I/O, file locking (databases, OS files, NFS shares). Use object storage when you need scale, an HTTP API and rich metadata (user uploads, data lakes, backups, ML datasets). Most mature stacks run both, side by side.

Key Stats

Fact Value Source
Amazon S3 consistency Strong read-after-write for new objects, overwrites and LIST — since Dec 1, 2020, at no extra cost AWS What's New
s3fs-fuse random writes "random writes or appends to files require rewriting the entire object" s3fs-fuse README → Limitations
goofys write support "only sequential writes supported"; no symlinks/hardlinks; cannot rename directories with >1000 children goofys README → Current Status
Mountpoint for Amazon S3 Does "not implement all the features of a POSIX file system" — no directory renaming, no symlinks, no edits to existing files awslabs/mountpoint-s3
RustFS license Apache 2.0 (no AGPL restrictions) rustfs/rustfs README
RustFS FUSE mount Not offered. The README's Feature & Status table lists no FUSE / POSIX mount driver — use a third-party S3 FUSE client against its S3 endpoint rustfs/rustfs README

What Is the Fundamental Difference Between Object and File Storage?

Dimension File Storage Object Storage
Data unit File (named byte sequence) Object (data + metadata + key)
Organization Hierarchical (directories/subdirectories) Flat (key namespace; / is cosmetic)
Access method POSIX (open/read/write/seek/close) HTTP REST API (PUT/GET/DELETE)
Mutability In-place (change bytes 100-200 without touching 1-99) Immutable (overwrite = new version/new object)
Metadata Fixed attributes (name, size, permissions, timestamps) Rich & extensible (custom key-value tags, content-type, etc.)
Scaling limit Millions of files (inode exhaustion, metadata perf) Billions+ of objects (distributed metadata)
Protocol NFS, SMB, POSIX local (ext4, xfs, zfs) S3 API (HTTP/HTTPS)
Consistency model Strong (reads see writes immediately) Strong read-after-write on AWS S3 since Dec 2020 — covers new objects, overwrites and LIST; S3-compatible systems vary
Typical latency Sub-millisecond (local) to milliseconds (NFS) Milliseconds (network round-trip)
Best for OS-level operations, databases, home dirs Unstructured data at scale, web/mobile apps, analytics

When Should You Use File Storage?

File storage is the right choice when your application (or OS) needs POSIX semantics:

Use Case 1: Operating System Files 

/etc/hosts
/var/log/syslog
/home/user/.bashrc
/tmp/processing_12345.tmp
Enter fullscreen mode Exit fullscreen mode

Your OS expects file storage. It uses open(), read(), write(), seek() — not HTTP PUT/GET. Don't fight this.

Use Case 2: Databases

PostgreSQL, MySQL, MongoDB, SQLite — they all expect block devices or file systems with:

  • Sub-millisecond random I/O (index lookups, page reads)
  • In-place mutation (UPDATE SET field = value changes specific bytes)
  • Strong consistency (ACID transactions depend on ordered fsync)
  • File locking (.lock files, advisory locks)

Object storage has millisecond-level latency and no in-place mutation. Databases on S3 perform terribly (with niche exceptions like Iceberg/Delta lakehouse patterns).

Use Case 3: Network File Sharing (NFS/SMB)

When multiple users/servers need shared access to the same files with familiar tools:

  • Design teams sharing Figma/Adobe files via SMB mount
  • Build servers sharing source code via NFS
  • Home directories in enterprise environments

These use cases need file-level permissions, directory browsing, and application transparency — all strengths of file storage.

Use Case 4: Small-Scale Applications (< 100K files, < 1TB)

For small datasets, file storage is simpler:

  • No API to learn
  • Familiar tools (ls, cp, rsync, grep)
  • Easy backups (tar, rsync)
  • Local access = fastest possible

When Should You Use Object Storage?

Object storage is the right choice when you need scale, simplicity of API, and rich metadata:

Use Case 1: User-Generated Content

Photos, videos, documents, uploads — the canonical object storage workload:

# User uploads photo
s3.put_object(
    Bucket="user-photos",
    Key=f"user-{user_id}/photo-{uuid}.jpg",
    Body=image_data,
    ContentType="image/jpeg",
    Metadata={"uploader": str(user_id), "camera": "iphone"}
)
Enter fullscreen mode Exit fullscreen mode

Scale from 1K to 100M objects without changing code. I have watched teams try to stretch a filesystem to that size; inode exhaustion is not a fun afternoon.

Use Case 2: Data Lake / Analytics

Parquet/Avro/CSV files for Spark, Trino, DuckDB:

s3://data-lake/bronze/events/year=2026/month=07/day=24/event-*.parquet
s3://data-lake/gold/daily_active_users.parquet
Enter fullscreen mode Exit fullscreen mode

Flat namespace, massive scale, accessed by query engines that speak S3 natively. This is where object storage dominates in 2026.

Use Case 3: Backup & Archive

Database dumps, VM snapshots, compliance records:

  • Immutable (versioning = accidental deletion protection)
  • Tiered (lifecycle policies move old data to cheap storage automatically)
  • Replicated (cross-region for DR)
  • Compliant (Object Lock for WORM retention)

File storage can do backups too, but at scale, object storage's tiering and replication features save significant cost and operational effort.

Use Case 4: Static Website / CDN Origin

S3 + CloudFront (or Cloudflare) is the standard pattern for serving static web content:

  • Objects = web assets (HTML, CSS, JS, images)
  • Global CDN = fast delivery everywhere
  • HTTPS + custom domain = zero-infrastructure frontend

Serving a global static site from an NFS mount is not something I'd want to run on-call for.

Use Case 5: Machine Learning & AI

Training data, model checkpoints, inference outputs:

  • Checkpointing: Write model state as object → resume from any saved point
  • Dataset versioning: Each dataset version = immutable object (reproducible training)
  • Feature stores: Parquet objects queried by ML frameworks via S3 API

ML workloads at scale (terabytes of training data) are almost always object-storage-backed in 2026.

Can You Mount S3 as a Filesystem?

What if you want S3's scale but need file-system semantics? I have been asked this in almost every object-storage migration. The honest answer is: you can, but the mount layer will lie to you in small ways. Here is what each project officially documents:

Tool Language What it does Officially documented limits
s3fs-fuse C++ Mounts an S3 bucket via FUSE on Linux/macOS/FreeBSD; preserves the native object format so aws s3 still works "random writes or appends to files require rewriting the entire object"; "no atomic renames of files or directories"; "no hard links"; "no coordination between multiple clients mounting the same bucket"
goofys Go A "Filey System" that "strives for performance first and POSIX second"; close-to-open consistency, no on-disk cache "only sequential writes supported"; "does not support symlink or hardlink"; "cannot rename directories with more than 1000 children"; "fsync is ignored" — and the last commit was June 2023, so treat it as low-maintenance
Mountpoint for Amazon S3 Rust AWS's own GA file client, tuned for high read throughput and sequential writes of new objects AWS states it is "probably not the right fit" for apps that use "directory renaming or symlinks" or "make edits to existing files (don't work on your Git repository or run vim in Mountpoint)"; support for non-AWS S3-compatible stores is limited
rclone mount Go Mounts any of rclone's 70+ backends, including any S3-compatible endpoint rclone's own docs warn the file system is not fully POSIX-compliant; behaviour depends on VFS cache mode

Own it first: RustFS does not ship a FUSE driver. Its README Feature & Status table covers S3 core, versioning, bucket replication, event notifications, bitrot protection, Swift/Keystone and Helm charts — no POSIX mount. If you want a mount, point one of the clients above at RustFS's S3 endpoint like you would at any other S3 service. Anyone telling you a "native RustFS mount" exists is reading a spec sheet that doesn't.

Performance reality: the translation layer is POSIX → HTTP, so each metadata operation becomes a network round trip. s3fs-fuse names this explicitly: "metadata operations such as listing directories have poor performance due to network latency." That's fine for bulk work — cp, tar, grep, feeding a training job. It is not fine for databases, build systems or anything doing high-IOPS random writes, because those turn into whole-object rewrites.

Which Should You Pick? A Decision Flowchart 

Do you need sub-millisecond random I/O?
├─ YES → File Storage (database, OS files)
│         (or block storage)
│
└─ NO → Do you need POSIX semantics (ls, chmod, flock)?
    ├─ YES → File Storage (NFS/SMB shares, source code)
    │
    └─ NO → Will you exceed 1M files/objects?
       ├─ YES → Object Storage (S3/S3-compatible)
       │         (photos, data lake, backups, ML)
       │
       └─ NO → Either works; pick the simpler tool
                 for your team's skill set
Enter fullscreen mode Exit fullscreen mode

TL;DR

  1. File storage = hierarchical, mutable, POSIX, scales to millions. Use it for OS files, databases, NFS shares, small datasets.
  2. Object storage = flat, immutable, HTTP API, scales to billions. Use it for user content, data lakes, backups/archives, static sites, ML data.
  3. The crossover point is usually scale. Under 100K files/1TB: file storage is simpler. Over that: object storage wins on operational cost.
  4. Databases always want file/block storage (not object). ML/analytics always want object storage (not file).
  5. S3 FUSE clients (s3fs-fuse, goofys, AWS Mountpoint, rclone mount) give you file-system access to S3 data — every one of them documents real POSIX gaps (no atomic renames, sequential-writes-only, no edits to existing files). Fine for bulk I/O, wrong for databases. RustFS itself ships no FUSE driver; use one of these against its S3 endpoint.
  6. You can use both. Most mature infrastructures I have worked with run file storage and object storage side-by-side. The cleanest architecture is usually "file storage for the OS and databases, object storage for everything else" rather than forcing one paradigm to cover both.

Need S3-compatible object storage you can run yourself? RustFS is Apache 2.0 licensed (no AGPL strings) and its README lists S3 core, versioning, bucket replication, event notifications, bitrot protection, multi-tenancy and Helm charts as Available; Lifecycle Management, Distributed Mode and KMS are still marked Under Testing — so plan accordingly. Try it in one command:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
Enter fullscreen mode Exit fullscreen mode

[sourced verbatim from the RustFS GitHub README — NOT EXECUTED IN CI]. Console on port 9001, default credentials rustfsadmin / rustfsadmin — change them before you expose anything. Binaries and the rc CLI: rustfs.com/download.

FAQ

Can I replace NFS with S3/object storage?

Sometimes. For plain file sharing I usually skip the FUSE shim and serve objects through a web UI or pre-signed URLs — one less POSIX lie to debug. If you really need a mount, read the limits first: s3fs-fuse has "no atomic renames of files or directories" and "no coordination between multiple clients mounting the same bucket"; goofys supports "only sequential writes"; Mountpoint refuses edits to existing files. Compilers, build systems, anything calling flock() — those stay on real file storage.

Which is faster, object storage or file storage?

It depends where the reader is sitting. A local NVMe filesystem wins for a single machine; object storage wins when you need a CDN in front of it. The question I ask is not "which is faster" but "which is fast enough at this distance". Databases need the local path. A photo served worldwide needs the CDN path. I ignore quoted millisecond figures unless they come with the test setup attached.

Can databases run on object storage?

Traditional OLTP databases — PostgreSQL, MySQL — no, not as their primary data directory. They need in-place mutation, ordered fsync and sub-millisecond random reads, none of which object storage provides. What does work, and works extremely well, is the lakehouse pattern: query engines such as DuckDB (via httpfs), Trino, Spark and ClickHouse (S3 table engine) read Parquet/ORC directly out of S3, and table formats like Apache Iceberg and Delta Lake add ACID semantics on top of immutable objects. Object storage is also the universal backup target for databases. So the accurate statement is: analytics on object storage, yes; transactional storage engine on object storage, no.

How do I migrate from file storage to object storage?

Gradually, and by workload rather than by directory. A path that works: (1) point all new workloads at S3 from day one; (2) move user-generated content first — uploads are the natural fit; (3) move analytics data next, as Parquet in a bucket queried by Spark/Trino/DuckDB; (4) leave the legacy file server on NFS/SMB and mirror it to object storage for DR and archive; (5) never move OS files. rclone sync handles filesystem-to-S3 copies against any S3-compatible endpoint, and aws s3 sync works for AWS. Budget for a coexistence period — both paradigms running side by side is the normal end state, not a failure.

Does RustFS provide a POSIX or FUSE mount?

No. As of the check date on this article, the RustFS GitHub README's Feature & Status table lists S3 Core Features, Upload/Download, Versioning, Logging, Event Notifications, K8s Helm Charts, Keystone Auth, Swift API, Bitrot Protection, Single Node Mode, Bucket Replication and Multi-Tenancy as Available, with Lifecycle Management, Distributed Mode and RustFS KMS marked Under Testing. There is no FUSE driver, no rustfs mount command and no POSIX mount feature anywhere in the README or on docs.rustfs.com. If you need a mount, run s3fs-fuse, goofys or rclone mount against the RustFS S3 endpoint on port 9000 — exactly as you would against any other S3-compatible service.

Sources

All claims above were checked against primary sources on 2026-08-07:

  1. Amazon S3 strong read-after-write consistency (new objects, overwrites, LIST; Dec 1, 2020) — https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-now-delivers-strong-read-after-write-consistency-automatically-for-all-applications/
  2. s3fs-fuse "Limitations" section (random writes rewrite the whole object, no atomic renames, no hard links, no multi-client coordination) — https://github.com/s3fs-fuse/s3fs-fuse
  3. goofys "Current Status" non-POSIX behaviours (sequential writes only, no symlink/hardlink, 1000-child rename cap, fsync ignored); last commit June 2023 — https://github.com/kahing/goofys
  4. Mountpoint for Amazon S3 POSIX caveats (no directory renaming, no symlinks, no edits to existing files) — https://github.com/awslabs/mountpoint-s3
  5. rclone mount documentation — https://rclone.org/commands/rclone_mount/
  6. RustFS license, Feature & Status table, quickstart command and default credentials — https://github.com/rustfs/rustfs
  7. RustFS installation documentation (no FUSE/POSIX mount path listed) — https://docs.rustfs.com/

Top comments (0)