Dead disks, no errors: how we keep failures away from users

Rabata is S3-compatible object storage running on our own hardware in the EU and the US. In this four-part series our engineers explain how the stack works under the hood. Part 3 covered how files are split into chunks with redundancy. This part is about what that redundancy is for: surviving failures without the client seeing them.

Reads: rebuild the chunk and keep streaming

If a data chunk is unavailable (a dead disk, a slow server, a corrupted file), the reader rebuilds it from the erasure codes and keeps streaming. The client sees a slightly slower read, not a failure.

The read also keeps the recovery set in memory. If a chunk turns out to be unavailable, the chunks already fetched don't have to be downloaded again.

Writes: switch disks mid-upload

On write the same property works the other way. If a chunk can't be written after several retries, the target disk is replaced with another one right in the middle of the upload. The request doesn't fail.

This is what makes our storage fault-tolerant. Disks fail and become unavailable from time to time, but the errors don't reach users.

Every disk is a scheduled resource

At any moment several kinds of operations compete for the same disk: user reads, user writes, background chunk verification, chunk repair and rebalancing. Left uncontrolled, background work would push up request latency for clients.

So every chunk request to a storage server carries its initiator: a user, repair, the verifier, rebalancing, other background work. In front of each disk sits a bounded priority queue with a concurrency limit, plus a rate limiter that can slow down a specific initiator.

Priorities are explicit and configured per operation type. A user read outranks a verifier read. Deletion is the lowest-priority work of all: we delete in the background, and a user's delete request only cleans up the database.

The disk is modeled as a scheduled resource with known consumers and their operations, not as a filesystem anyone writes to whenever they like.

Hardware moves through explicit states

Servers and disks go through an explicit state machine:

State	Chunks written / read	What happens
Normal	Yes / yes	Hardware works as expected
Overload	Less likely / yes	Write load is reduced until metrics return to acceptable bounds
No metrics	No / no	The system has received no metrics from the hardware for a long time
Out of use	No / no	Metrics have been out of bounds for a very long time; an alert fires
Fault	No / no	The hardware is unfit for use; all its chunks are queued for repair; an alert fires
Maintenance	No / no	Returned to service manually only
Decommissioning	—	All chunks are queued for repair; once done, the hardware is removed automatically

States are derived from observed metrics, not set by hand. A dedicated control-plane service moves hardware between states and also computes disk weights for placement from free space and load.

Two ways a repair starts

1. Scrubbing. A background process runs on every storage server. It walks every chunk on every disk, reads it directly from the disk with O_DIRECT, and checks its integrity against a stored hash. Hidden corruption is found by scanning, not by a user who happened to read a bad chunk.
2. Disk fault. The control plane moves a disk to Fault and starts a job to repair all of its chunks. The job is split into smaller ones, which in turn schedule repairs of specific chunks.

Questions about any of it? Our engineers read the comments.

Rabata S3: S3-compatible storage at $0.006/GB/month with no API request fees. Change the endpoint and keys, keep your code.
`