Rabata S3 layer never stores a single byte of your data

Rabata is S3-compatible object storage running on our own hardware in the EU and the US. We built the stack ourselves, from the S3 API down to how bytes land on disk. In this four-part series our engineers explain how it works: the trade-offs we made, what they cost, and why. Part 1 is the map. Parts 2–4 go deep on metadata, redundancy and failure handling.

Three services, three jobs

Our storage is three independent services. Each has its own repository, its own databases and its own release cycle.

• s3-front is everything that makes up the S3 protocol: buckets, keys, versions, multipart uploads, ownership, ACLs and policies, request signing, encryption, usage accounting. It stores S3 metadata only. Not a single byte of object data.

• dstorage is distributed, fault-tolerant file storage: servers, disks, files and chunks. On upload a file is split into chunks with redundancy, cheap to keep but able to survive a server or disk failure. On download it is reassembled from them. Lost data is restored from chunk copies or from erasure codes such as Reed–Solomon. Chunk placement across servers and disks lives here too.

• IAM is authentication and authorization: accounts, IAM users, access keys, account state. It is out of scope for this series.

One integer between metadata and bytes

The split between s3-front and dstorage is deliberate. A file in dstorage is identified by a single 64-bit integer. That integer is the only thing s3-front keeps from dstorage.

The entire interaction between them is six operations:

1. Reserve a file ID.
2. Upload a byte stream.
3. Read a byte range.
4. Read a concatenation of ranges from several files, so a multipart object is served in one network request instead of one per part.
5. Copy one byte stream into another.
6. Delete a file.

What the split buys us

dstorage exposes only a file: a byte stream with an ID. So inside it we can change the redundancy scheme, chunk size, disk layout and placement policy without touching the S3 layer.

The reverse holds too. S3 semantics (versioning rules, conditional requests, listing order) never leak into the storage layer. Each service is tested and rolled out independently.

Consistency without distributed transactions

Neither service has distributed transactions. Yet each has to keep two independent stores in agreement. For s3-front, that is its metadata and the files in dstorage. For dstorage, its metadata and the bytes on disks. Both solve it the same way.

Every write follows one order:

1. Record intent. Before a single byte is written, the operation records that it is about to write: a timestamped marker in the same shard that will hold the final metadata.
2. Write data. Bytes go over the network to the disks.
3. Commit metadata. Only once the bytes are durably written is metadata committed, atomically, in a single script.

Metadata is committed last, so it never points at bytes that aren't there. And because intent is recorded first, no bytes are ever left untracked: if a write dies halfway, the marker says what to clean up.

Orphaned bytes are removed asynchronously, the same way in both systems. Write and delete paths put records into queues. Periodic jobs pick them up from the database and move them to a deletion queue.

Next in the series:

• Part 2: why we don't hash S3 keys, and how we get atomic S3 operations without transactions.
• Part 3: how we pick redundancy per file, and why 1.4× beats 3× for most data.
• Part 4: what happens when a disk dies in the middle of your upload.

Rabata S3: S3-compatible storage at $0.006/GB/month with no API request fees. Change the endpoint and keys, keep your code.