Rabata is S3-compatible object storage running on our own hardware in the EU and the US. In this four-part series our engineers explain how the stack works under the hood. Part 1 covered the overall architecture. This part is about s3-front, the service that holds all S3 metadata.
Why Redis
All S3 metadata lives in Redis. So does metadata in our other services. The main argument is speed: request latency is measured in microseconds, not in the milliseconds of relational databases. Object metadata is small, uniform, and read far more often than written.
Redis also comes with constraints. Everything lives in memory, so data volume is bounded by RAM. Redis is single-threaded, so it scales through sharding and replicas. Three more constraints define our whole data model.
No transactions: one script per S3 operation
Redis has no transactions, or more precisely, no atomicity across commands. There are only Lua scripts and optimistic locking via WATCH.
A single S3 operation touches several metadata structures at once: the object record, for multipart objects the records of their parts, the latest-version pointer, the bucket's ordered listing index and the bucket's usage counters.
Doing that from the client would mean a chain of network calls with no atomicity and no way to roll back. So every S3 operation is wrapped in a script executed inside Redis. It reads what it needs, checks preconditions, applies all mutations and returns the result. The unit of atomicity is the script. The boundary of that atomicity is one shard.
No cross-slot commands: every object lands in one shard
A single command can't touch keys from different slots. So an operation that can't be expressed within one shard can't be atomic.
The data model is built so that any operation on one object (put, get, delete, any multipart step) lands in exactly one shard by construction. This is not an optimization. It is the reason the model looks the way it does.
No secondary indexes: every access pattern is built by hand
There is no query planner. Each access pattern S3 requires gets its own structure, written synchronously by the same script that writes the object:
- an ordered per-bucket index for listings;
- a pointer to resolve the latest version;
- an offset index to find the parts of a multipart object.
The obvious answer: hash the key
In Redis Cluster the unit of sharding is the slot, and there are always exactly 16,384 of them. Every metadata key belongs to exactly one slot. The objects of each bucket live across several slots, and an object is routed to a slot by its key.
The obvious way to route is hashing: hash the object key, get the slot. It gives near-perfect load distribution and needs no coordination.
But it makes ListObjects inefficient. S3 requires results in lexicographic order, with prefixes, delimiters and resumable pagination. With hash sharding, every listing fans out to all shards. Results then have to be merge-sorted, and most of what was read is thrown away. The cost grows with the number of shards, not with the size of the response.
And since the number of objects in a bucket is unbounded, keeping a whole bucket on one shard is not an option either.
What we do instead: range sharding
Buckets are sharded through an explicit mapping. The object key space is split into contiguous key ranges, and one bucket can span several shards.
- A request for an object goes to the slot whose range contains its key.
- A listing starts at the slot that contains the start key and walks forward in order. It stops as soon as the next slot's range falls outside the requested prefix.
The price: hot ranges
Range sharding has a weak spot: hot prefixes, or keys written by date, where every write lands at the end of the range. Range sharding turns that skew into a hot shard, so placement has to be managed actively.
Today per-slot throughput leaves us a large margin. If that changes, there is a middle ground: hash the key first and give each bucket a range of hashes. A bucket then spreads over a controlled subset of shards rather than all of them.
New buckets are placed by weighted random choice across slots. Weights are recalculated periodically from the number of keys per slot, so emptier slots are more likely to get new buckets.
Multi-region: only bucket names are global
Exactly one thing must be globally consistent. Bucket names in S3 are globally unique, so the registry “name → owner and region” is shared across regions. Everything else (object metadata, usage accounting, control state) is strictly regional.
A request that hits the wrong region is not an error. The registry says which region owns the bucket, and the request is proxied there over an internal API. Global consistency is limited to a small, rarely changing, cacheable dataset. The expensive parts of the system never pay for cross-region coordination.
For contrast: a layer that doesn't need order
The storage layer, dstorage, also keeps its metadata in Redis Cluster, with the same constraints but different sharding. Files are addressed by ID and there are no listings, so order doesn't matter.
New file IDs are handed out across slots round-robin, and the slot is encoded in the ID itself: it is derived arithmetically, with no index lookup. Load spreads evenly, and hot ranges simply don't occur.
The series
• Part 1: our S3 layer never stores a single byte of your data.
• Part 3: how we pick redundancy per file.
• Part 4: what happens when a disk dies in the middle of your upload.
Rabata S3: S3-compatible storage at $0.006/GB/month with no API request fees. Change the endpoint and keys, keep your code.
Top comments (0)