DEV Community: Rabata

Dead disks, no errors: how we keep failures away from users

Rabata — Thu, 29 Oct 2026 13:17:17 +0000

Rabata is S3-compatible object storage running on our own hardware in the EU and the US. In this four-part series our engineers explain how the stack works under the hood. Part 3 covered how files are split into chunks with redundancy. This part is about what that redundancy is for: surviving failures without the client seeing them.

Reads: rebuild the chunk and keep streaming

If a data chunk is unavailable (a dead disk, a slow server, a corrupted file), the reader rebuilds it from the erasure codes and keeps streaming. The client sees a slightly slower read, not a failure.

The read also keeps the recovery set in memory. If a chunk turns out to be unavailable, the chunks already fetched don't have to be downloaded again.

Writes: switch disks mid-upload

On write the same property works the other way. If a chunk can't be written after several retries, the target disk is replaced with another one right in the middle of the upload. The request doesn't fail.

This is what makes our storage fault-tolerant. Disks fail and become unavailable from time to time, but the errors don't reach users.

Every disk is a scheduled resource

At any moment several kinds of operations compete for the same disk: user reads, user writes, background chunk verification, chunk repair and rebalancing. Left uncontrolled, background work would push up request latency for clients.

So every chunk request to a storage server carries its initiator: a user, repair, the verifier, rebalancing, other background work. In front of each disk sits a bounded priority queue with a concurrency limit, plus a rate limiter that can slow down a specific initiator.

Priorities are explicit and configured per operation type. A user read outranks a verifier read. Deletion is the lowest-priority work of all: we delete in the background, and a user's delete request only cleans up the database.

The disk is modeled as a scheduled resource with known consumers and their operations, not as a filesystem anyone writes to whenever they like.

Hardware moves through explicit states

Servers and disks go through an explicit state machine:

State	Chunks written / read	What happens
Normal	Yes / yes	Hardware works as expected
Overload	Less likely / yes	Write load is reduced until metrics return to acceptable bounds
No metrics	No / no	The system has received no metrics from the hardware for a long time
Out of use	No / no	Metrics have been out of bounds for a very long time; an alert fires
Fault	No / no	The hardware is unfit for use; all its chunks are queued for repair; an alert fires
Maintenance	No / no	Returned to service manually only
Decommissioning	—	All chunks are queued for repair; once done, the hardware is removed automatically

States are derived from observed metrics, not set by hand. A dedicated control-plane service moves hardware between states and also computes disk weights for placement from free space and load.

Two ways a repair starts

Scrubbing. A background process runs on every storage server. It walks every chunk on every disk, reads it directly from the disk with O_DIRECT, and checks its integrity against a stored hash. Hidden corruption is found by scanning, not by a user who happened to read a bad chunk.
Disk fault. The control plane moves a disk to Fault and starts a job to repair all of its chunks. The job is split into smaller ones, which in turn schedule repairs of specific chunks.

Questions about any of it? Our engineers read the comments.

Rabata S3: S3-compatible storage at $0.006/GB/month with no API request fees. Change the endpoint and keys, keep your code.
`

Same durability, less than half the disk: how we pick redundancy per file

Rabata — Thu, 29 Oct 2026 13:03:09 +0000

Rabata is S3-compatible object storage running on our own hardware in the EU and the US. In this four-part series our engineers explain how the stack works under the hood. Part 1 and Part 2 covered the architecture and the S3 metadata layer. This part is about dstorage, the service that stores the bytes.

Files, recovery sets, chunks

A file is a byte stream with a numeric ID. On write it is split into recovery sets, and each recovery set consists of several chunks. Every chunk in a recovery set is placed on a different server.

A chunk's address is a triple: (file ID, chunk index, disk ID). File metadata (size, redundancy scheme, the map of chunks to disks) lives in a sharded metadata store. Chunk bytes never go there. The metadata store holds only what's needed to find a chunk and to know the state of the hardware.

How many extra bytes

Disks die. Servers reboot. A filesystem sometimes returns something other than what was written. Storage has to survive all of this without losing data, which means storing more bytes than the file itself takes. The only question is how many more.

The simplest answer is several full copies on different servers. Three copies survive the loss of any two disks, but take three times the space of the data.

Erasure codes solve the same problem cheaper. The file is cut into N data chunks, and M more parity chunks are computed from them arithmetically. All N + M go to different servers. The trick is that the original data can be restored from any N chunks of the set: lose two, and the missing ones are recomputed from the rest.

The space overhead drops from (M + 1)× to (N + M) / N, and durability stays the same: the set survives the loss of any M chunks.

Small files pay a per-chunk tax

We pay for the saved space with computation on every write and every chunk repair. On small files we also pay per-chunk overhead. Erasure coding a 3 KiB file gives chunks of a few hundred bytes. For each such chunk, the filesystem write, the metadata record, the network call and the I/O operation outweigh what the chunk protects.

So the redundancy scheme is chosen for each file based on its size. Two schemes are configured today:

Scheme	Used for	Layout	Space	Survives
Copy	Small files	The file + 2 copies, all on different servers	3×	Loss of any 2 copies
Reed–Solomon 5+2	Everything above the threshold	5 data + 2 parity chunks per recovery set	1.4×	Loss of any 2 of 7 chunks

Copy costs more space, but a small file is one chunk instead of seven, and nothing has to be computed on read.

Schemes are stored by ID, not by value

File metadata records which configuration the file was written with. New configurations are added rarely.

If we want a new scheme, or new parameters for an existing one (a different number of data or parity chunks, a different chunk size), we don't need a migration. Old files are always read with the scheme they were written with. A new redundancy policy rolls out as a config change and affects new writes only.

Chunk count matters more than byte count

The last group of a file rarely divides evenly. Instead of an incomplete group of tiny chunks, the remainder is appended to the previous group. This keeps chunk sizes in a sane range regardless of file size.

That matters because the number of chunks, not the number of bytes, defines metadata volume, the number of I/O operations and the cost of repairing a chunk.

No per-chunk index

There is no index that maps byte offsets to chunks. The layout is computed. Given a file's size and redundancy scheme, the position of any byte (which recovery set, which chunk, which offset inside the chunk) can be calculated. Reads and repairs of damaged or unavailable chunks use the same code.

Streaming writes with bounded memory

Writes are streamed, but erasure coding needs a full parity group in memory to compute the parity chunks. Chunks of a recovery set are accumulated and flushed once parity is computed and the chunks are uploaded to storage servers.

Memory use is bounded by one parity group (group width × chunk size), whether the file is 1 MiB or 1 TiB. And the parallelism that matters is preserved: the chunks of a group go to different disks on different servers at the same time.

Rabata S3: S3-compatible storage at $0.006/GB/month with no API request fees. Change the endpoint and keys, keep your code.
`

No hashing, no fan-out: how we shard S3 metadata for ListObjects

Rabata — Tue, 27 Oct 2026 13:56:43 +0000

Rabata is S3-compatible object storage running on our own hardware in the EU and the US. In this four-part series our engineers explain how the stack works under the hood. Part 1 covered the overall architecture. This part is about s3-front, the service that holds all S3 metadata.

Why Redis

All S3 metadata lives in Redis. So does metadata in our other services. The main argument is speed: request latency is measured in microseconds, not in the milliseconds of relational databases. Object metadata is small, uniform, and read far more often than written.

Redis also comes with constraints. Everything lives in memory, so data volume is bounded by RAM. Redis is single-threaded, so it scales through sharding and replicas. Three more constraints define our whole data model.

No transactions: one script per S3 operation

Redis has no transactions, or more precisely, no atomicity across commands. There are only Lua scripts and optimistic locking via WATCH.

A single S3 operation touches several metadata structures at once: the object record, for multipart objects the records of their parts, the latest-version pointer, the bucket's ordered listing index and the bucket's usage counters.

Doing that from the client would mean a chain of network calls with no atomicity and no way to roll back. So every S3 operation is wrapped in a script executed inside Redis. It reads what it needs, checks preconditions, applies all mutations and returns the result. The unit of atomicity is the script. The boundary of that atomicity is one shard.

No cross-slot commands: every object lands in one shard

A single command can't touch keys from different slots. So an operation that can't be expressed within one shard can't be atomic.

The data model is built so that any operation on one object (put, get, delete, any multipart step) lands in exactly one shard by construction. This is not an optimization. It is the reason the model looks the way it does.

No secondary indexes: every access pattern is built by hand

There is no query planner. Each access pattern S3 requires gets its own structure, written synchronously by the same script that writes the object:

an ordered per-bucket index for listings;
a pointer to resolve the latest version;
an offset index to find the parts of a multipart object.

The obvious answer: hash the key

In Redis Cluster the unit of sharding is the slot, and there are always exactly 16,384 of them. Every metadata key belongs to exactly one slot. The objects of each bucket live across several slots, and an object is routed to a slot by its key.

The obvious way to route is hashing: hash the object key, get the slot. It gives near-perfect load distribution and needs no coordination.

But it makes ListObjects inefficient. S3 requires results in lexicographic order, with prefixes, delimiters and resumable pagination. With hash sharding, every listing fans out to all shards. Results then have to be merge-sorted, and most of what was read is thrown away. The cost grows with the number of shards, not with the size of the response.

And since the number of objects in a bucket is unbounded, keeping a whole bucket on one shard is not an option either.

What we do instead: range sharding

Buckets are sharded through an explicit mapping. The object key space is split into contiguous key ranges, and one bucket can span several shards.

A request for an object goes to the slot whose range contains its key.
A listing starts at the slot that contains the start key and walks forward in order. It stops as soon as the next slot's range falls outside the requested prefix.

The price: hot ranges

Range sharding has a weak spot: hot prefixes, or keys written by date, where every write lands at the end of the range. Range sharding turns that skew into a hot shard, so placement has to be managed actively.

Today per-slot throughput leaves us a large margin. If that changes, there is a middle ground: hash the key first and give each bucket a range of hashes. A bucket then spreads over a controlled subset of shards rather than all of them.

New buckets are placed by weighted random choice across slots. Weights are recalculated periodically from the number of keys per slot, so emptier slots are more likely to get new buckets.

Multi-region: only bucket names are global

Exactly one thing must be globally consistent. Bucket names in S3 are globally unique, so the registry “name → owner and region” is shared across regions. Everything else (object metadata, usage accounting, control state) is strictly regional.

A request that hits the wrong region is not an error. The registry says which region owns the bucket, and the request is proxied there over an internal API. Global consistency is limited to a small, rarely changing, cacheable dataset. The expensive parts of the system never pay for cross-region coordination.

For contrast: a layer that doesn't need order

The storage layer, dstorage, also keeps its metadata in Redis Cluster, with the same constraints but different sharding. Files are addressed by ID and there are no listings, so order doesn't matter.

New file IDs are handed out across slots round-robin, and the slot is encoded in the ID itself: it is derived arithmetically, with no index lookup. Load spreads evenly, and hot ranges simply don't occur.

Rabata S3: S3-compatible storage at $0.006/GB/month with no API request fees. Change the endpoint and keys, keep your code.

Zero bytes of your data: how our S3 layer keeps only metadata

Rabata — Tue, 27 Oct 2026 13:20:49 +0000

Rabata is S3-compatible object storage running on our own hardware in the EU and the US. We built the stack ourselves, from the S3 API down to how bytes land on disk. In this four-part series our engineers explain how it works: the trade-offs we made, what they cost, and why. Part 1 is the map. Parts 2–4 go deep on metadata, redundancy and failure handling.

Three services, three jobs

Our storage is three independent services. Each has its own repository, its own databases and its own release cycle.

s3-front is everything that makes up the S3 protocol: buckets, keys, versions, multipart uploads, ownership, ACLs and policies, request signing, encryption, usage accounting. It stores S3 metadata only. Not a single byte of object data.
dstorage is distributed, fault-tolerant file storage: servers, disks, files and chunks. On upload a file is split into chunks with redundancy, cheap to keep but able to survive a server or disk failure. On download it is reassembled from them. Lost data is restored from chunk copies or from erasure codes such as Reed–Solomon. Chunk placement across servers and disks lives here too.
IAM is authentication and authorization: accounts, IAM users, access keys, account state. It is out of scope for this series.

One integer between metadata and bytes

The split between s3-front and dstorage is deliberate. A file in dstorage is identified by a single 64-bit integer. That integer is the only thing s3-front keeps from dstorage.

The entire interaction between them is six operations:

Reserve a file ID.
Upload a byte stream.
Read a byte range.
Read a concatenation of ranges from several files, so a multipart object is served in one network request instead of one per part.
Copy one byte stream into another.
Delete a file.

What the split buys us

dstorage exposes only a file: a byte stream with an ID. So inside it we can change the redundancy scheme, chunk size, disk layout and placement policy without touching the S3 layer.

The reverse holds too. S3 semantics (versioning rules, conditional requests, listing order) never leak into the storage layer. Each service is tested and rolled out independently.

Consistency without distributed transactions

Neither service has distributed transactions. Yet each has to keep two independent stores in agreement. For s3-front, that is its metadata and the files in dstorage. For dstorage, its metadata and the bytes on disks. Both solve it the same way.

Every write follows one order:

1. Record intent. Before a single byte is written, the operation records that it is about to write: a timestamped marker in the same shard that will hold the final metadata.
2. Write data. Bytes go over the network to the disks.
3. Commit metadata. Only once the bytes are durably written is metadata committed, atomically, in a single script.

Metadata is committed last, so it never points at bytes that aren't there. And because intent is recorded first, no bytes are ever left untracked: if a write dies halfway, the marker says what to clean up.

Orphaned bytes are removed asynchronously, the same way in both systems. Write and delete paths put records into queues. Periodic jobs pick them up from the database and move them to a deletion queue.

Rabata S3: S3-compatible storage at $0.006/GB/month with no API request fees. Change the endpoint and keys, keep your code.

Cloud storage credits for Gen-AI startups: what's real, what's not, and what we're offering

Rabata — Tue, 05 May 2026 10:15:44 +0000

If you're building a Gen-AI startup, you've probably already seen the headlines. $100K here, $350K there. Every major cloud has a startup program and they all sound generous until you read the fine print.

This is a breakdown of what those programs actually look like in practice: the caps, the catches, and the pricing cliffs nobody puts in the headline. And at the end, what we're offering at Rabata and why.

What programs actually look like

Every major cloud has a startup program. The numbers look generous on paper.

AWS Activate offers up to $100K — but the $100K tier requires affiliation with an approved VC or accelerator. And **"up to" **is doing a lot of work in that sentence. In practice, AWS evaluates your stage, traction, and provider relationship and most early-stage teams walk away with $5K–$25K, not the headline number. Even then, AWS's own support forum acknowledges they have "internal requirements beyond what's publicly listed." Founders get rejected with no explanation. Credits expire in 12–24 months and AWS will not extend them, even if you have unused balance.

One subtle trap: the Free Tier silently disappears the moment you set up AWS Organizations (their own recommended multi-account structure) with no warning.

Google for Startups offers up to $350K for AI startups. Year 1 covers 100% of GCP usage up to $250K — that part is real. Year 2 is 20% off your usage up to $100K. To redeem the full $100K in Year 2, you'd need to be spending $500K/year on GCP. For most early-stage teams, that number isn't there yet.

Cloudflare for Startups goes up to $250K, but the top tier requires Tier-1 VC backing. More importantly for storage-heavy workloads: R2 (their S3 alternative) is capped at $10K of credit usage regardless of your headline tier. If you're storing model checkpoints, training data, or generated video at any scale, you'll hit that in weeks.

These programs exist for good reasons and some of them are worth applying to — especially for compute and managed AI services. But they're designed around platform lock-in: the clock starts on day one, the credits are capped on the things you actually need, and the pricing after credits run out is full rate.

The trap nobody talks about upfront

Before you pick any cloud provider (credit program or not) run the egress math first.

AWS charges $0.09/GB for data transfer out. That doesn't sound catastrophic until you're moving terabytes of training data, model outputs, or video assets regularly. At scale, egress alone can exceed your storage costs.

The worse part: once you're on AWS and your data is there, leaving gets expensive too. Migrating out means paying egress on everything you move. So you end up in a position where staying is costly and leaving is also costly. That's not an accident — it's how the economics are designed.

This applies beyond AWS. Check egress pricing on day one, not after you've built your pipeline around a provider. The credit offer is the entry point; the pricing model is what you'll actually live with.

Why we built our grant

Gen-AI startups are data-heavy in a way most software startups aren't. Training datasets, model checkpoints, fine-tuning runs, inference logs, generated outputs — storage compounds fast, and it doesn't stop after your free credits expire.

Because at Rabata we control our infrastructure costs, we can price differently from resellers and hyperscalers: $0.01/GB/month, free ingress, $0.01/GB egress, no per-request fees.

At that price, **$100K in Rabata credits is roughly 833TB of active storage for a year. **The same $100K on AWS S3 Standard gets you around 250TB, before egress. If your pipeline moves data between services, that gap widens further.

That's not a marketing trick — it's what happens when you don't resell AWS underneath your product.

What the Rabata grant is

$100K in Rabata credits
Covers S3-compatible object storage
2-year duration — not 12 months, not 90 days
No equity. No VC requirement. No exclusivity.
Short application form. We respond within 24–48 hours.

*Eligibility: * you're building a Gen-AI product and have real storage needs. That's it.

Apply here: https://rabata.io/grant-application

We benchmarked 10+ S3 providers — here's what the numbers actually show

Rabata — Fri, 01 May 2026 13:23:12 +0000

With so many S3-compatible providers on the market, it can be difficult to understand which one truly fits your workload.

To keep things objective, we used MinIO warp v1.0.7, a widely adopted open-source benchmarking tool. Tests included upload, download, mixed workloads, small object throughput, and large object handling. All benchmarks were run under identical conditions on a Debian 13 VM with 16 GB RAM, located in US-East-1, using 8 concurrent threads and consistent object sizes.

Our methodology is fully reproducible — every parameter and setup detail is publicly available. You can find the complete benchmark methodology and replication guide with step-by-step instructions to run your own tests. All pricing information reflects rates publicly available as of October 1, 2025.

Here's the summary:

Performance Comparison" width="800" height="394">
A few things that surprised us:

Mixed operations (simultaneous reads, writes, metadata) is the most realistic measure of real-world performance — and results vary wildly
Cloudflare R2 has zero egress fees but came in last on mixed ops (44 Mbit/s) and small objects (42 obj/s)
Hetzner's numbers look low — but that's because we tested from US-East-1 and they're EU-only, so transatlantic latency skews the results
Backblaze B2 dominates on downloads but showed timeouts during large-object tests

For full per-provider breakdowns and detailed analysis: https://rabata.io/s3-comparison