DEV Community

Cover image for Using S3 as Your Disaster Recovery Target: A Practical Guide
Ethan Carter
Ethan Carter

Posted on

Using S3 as Your Disaster Recovery Target: A Practical Guide

#s3 #disasterrecovery #backup #rustfs

Using S3 as Your Disaster Recovery Target: A Practical Guide

S3 works as a disaster-recovery target because your data is already there, it is redundant by design, and cross-region replication is a configuration change rather than a new pipeline. The case for bothering is in the outage economics: Uptime Institute's 2022 Outage Analysis found that over 60% of failures now cost at least $100,000 — up from 39% in 2019 — and the share costing more than $1 million climbed from 11% to 15%. Yet when I audit DR plans, S3 is consistently the most underused asset in the room.

Key Stats

Metric Figure Source
Failures costing ≥ $100,000 60%+ (was 39% in 2019) Uptime Institute, 2022 Outage Analysis
Failures costing ≥ $1M 15% (was 11%) Uptime Institute, 2022 Outage Analysis
Orgs hit by a serious/severe outage in 3 years 1 in 5 Uptime Institute, 2022 Outage Analysis
Public outages lasting > 24 hours (2021) ~30% (was 8% in 2017) Uptime Institute, 2022 Outage Analysis
Major outages caused by human error ~40% of orgs; 85% trace to procedure failures Uptime Institute, 2022 Outage Analysis
AWS S3 designed durability 99.999999999% (11 nines) AWS S3 official documentation

That last row on human error is the one worth sitting with: the failure mode is rarely "the storage broke." It is that nobody had run the restore.

I have audited enough DR plans to notice a pattern: the teams that survive outages are not the ones with the most elegant architecture; they are the ones that actually tested a restore. S3-compatible storage is just the cheapest place to start that habit.

Why does S3 work well as a DR target?

It's Already There

Most applications I look at already write something to S3, even if the team does not think of it as primary storage:

  • User-uploaded files (photos, documents, media)
  • Log files and audit trails
  • Database backups (mysqldump, pg_dump, volume snapshots)
  • ML model artifacts and datasets
  • Configuration backups

If this data is already in S3, making it DR-capable is mostly a configuration change, not a new pipeline.

It's Durable by Design

S3 (and serious S3-compatible implementations) stores data redundantly:

Implementation Stated Durability Mechanism
AWS S3 99.999999999% (11 nines), designed for Redundant storage across multiple devices in ≥3 Availability Zones
Backblaze B2 99.999999999% (11 nines), annual Reed-Solomon erasure coding
RustFS No published nines figure README lists Bitrot Protection and Bucket Replication as ✅ Available; Distributed Mode is 🚧 Under Testing
MinIO No published nines figure Erasure coding or replication, configurable

I am deliberately not inventing a durability number for RustFS or MinIO. Neither project publishes an audited nines figure, and a self-hosted cluster's real durability depends on your disk count, erasure set width and failure domains — not on the vendor's marketing page.

Single-disk failure = zero data loss. Single-node failure (in clustered setups) = zero data loss. This is better than most on-premises databases achieve out of the box.

It's Cheap (Relative to Alternatives)

DR is insurance — nobody wants to overpay for it. Concretely: AWS S3 Standard list price is $0.023/GB-month, so parking 10 TB of DR backups costs about $236/month. A warm-standby database instance sized for the same workload, plus its attached block storage, generally lands in four figures a month. I am not going to quote you a tidy multiple, because the ratio swings hard with instance class, retention and egress — run the numbers for your own shape.

Which DR architecture pattern should you use?

Pattern 1: Backup-to-S3 (What I recommend first) 

Production DB ──[daily dump]──▶ S3 Bucket (primary region)
                                    │
                              Cross-region replicate
                                    │
                                    ▶
                              S3 Bucket (DR region)
Enter fullscreen mode Exit fullscreen mode

How it works:

  1. Nightly database dump → compressed file → PUT to S3
  2. S3 Cross-Region Replication (CRR) copies to DR region
  3. If primary region fails: spin up DB in DR region → restore from S3

RTO (Recovery Time Objective): 1-4 hours (depends on DB size + restore speed)
RPO (Recovery Point Objective): Up to 24 hours (backup frequency)

This is not exciting architecture, but it is the pattern that saves most teams. Get backups into a second region before you worry about streaming replication.

Tools: aws s3 sync, rclone, database-native S3 backup tools (pg_backrest, mysqldump + pipe)

Pattern 2: Continuous Log Shipping (Better RPO) 

Production DB WAL/Binlog ──[stream]──▶ S3 (primary)
                                         │
                                   CRR / Custom forwarder
                                         │
                                         ▶
                                   S3 (DR region)
                                         │
                                   [Continuous restore]
                                         ▶
                                   Standby DB (DR region)
Enter fullscreen mode Exit fullscreen mode

How it works:

  1. Write-Ahead Logs (PostgreSQL) or Binlogs (MySQL) stream to S3 continuously
  2. Standby DB in DR region applies logs in near-real-time
  3. If primary fails: promote standby (seconds to minutes of RPO)

RTO: Minutes (standby is already running, just needs promotion)
RPO: Seconds to minutes (depends on log shipping lag)

Tools: WAL-G (PostgreSQL), pgBackRest, MySQL binlog-to-s3 tools, Debezium (CDC)

Pattern 3: Active-Active with S3 as Source of Truth (Avoid unless you need it) 

Region A App ◄──── S3 (shared, replicated) ───► Region B App
                    │
              [Both regions read/write to same S3]
              [App-level conflict resolution required]
Enter fullscreen mode Exit fullscreen mode

How it works:

  1. Both regions' applications read/write to the same S3 bucket(s)
  2. Conflict resolution is your application's job — object storage does not merge concurrent writes for you
  3. If Region A fails: Region B continues serving with zero RPO

RTO: Zero (automatic)
RPO: Zero (both regions always current)

Complexity: High. I have seen this pattern look simple on a whiteboard and turn into weeks of conflict-resolution bugs in production. Use it only when zero RPO is a hard business requirement, not because it sounds modern.

Implementing Pattern 1: The Minimal Viable DR Plan

Step 1: Identify Critical Data

Not all data needs DR protection. Classify yours:

Tier Data Type Example DR Requirement
T0 (Critical) User data, financial transactions User uploads, payment records RPO < 1hr, RTO < 1hr
T1 (Important) Business operational data Logs, configs, ML models RPO < 24hr, RTO < 4hr
T2 (Useful) Analytics, historical Aggregated data, old backups RPO < 7 days, RTO < 24hr
T3 (Disposable) Cache, temp files Session stores, build artifacts Best effort / none

Focus DR effort on T0 and T1. T2 and T3 are nice-to-have.

Step 2: Set Up Automated Backups to S3 

#!/bin/bash
# dr-backup.sh — Daily backup to S3
DATE=$(date +%Y-%m-%d)
BUCKET="dr-backups-primary"

# PostgreSQL backup
pg_dump -Fc production_db | gzip | aws s3 cp - "s3://$BUCKET/postgres/$DATE/db.dump.gz"

# Application data sync
aws s3 sync /data/app-uploads "s3://$BUCKET/uploads/$DATE/" --delete

# Config backup
tar czf - /etc/myapp/config | aws s3 cp - "s3://$BUCKET/config/$DATE/config.tar.gz"

echo "[$(date)] DR backup complete" >> /var/log/dr-backup.log
Enter fullscreen mode Exit fullscreen mode

Schedule via cron (every 6 hours for T0, daily for T1):

0 */6 * * * /opt/scripts/dr-backup.sh    # T0: Every 6 hours
0 2 * * * /opt/scripts/dr-backup-full.sh # T1: Daily at 2 AM
Enter fullscreen mode Exit fullscreen mode

Step 3: Configure Cross-Region Replication

On AWS. Replication requires versioning on both buckets, and Role is a required top-level field in the replication configuration — a lot of copy-pasted snippets omit it and fail with InvalidRequest.

# [sourced from https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-replication.html, NOT EXECUTED IN CI]

# Prerequisite: versioning on BOTH source and destination
aws s3api put-bucket-versioning \
  --bucket dr-backups-primary \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-versioning \
  --bucket dr-backups-dr \
  --versioning-configuration Status=Enabled

# CRR rule ("Role" is REQUIRED — the IAM role S3 assumes to replicate)
aws s3api put-bucket-replication \
  --bucket dr-backups-primary \
  --replication-configuration '{
    "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
    "Rules": [{
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Enabled" },
      "Filter": { "Prefix": "" },
      "Destination": {
        "Bucket": "arn:aws:s3:::dr-backups-dr",
        "StorageClass": "STANDARD"
      }
    }]
  }'
Enter fullscreen mode Exit fullscreen mode

On self-hosted S3. RustFS lists Bucket Replication as ✅ Available in its README Feature & Status table. If you want a scheduled, engine-agnostic copy between two independent clusters — which is what most self-hosted DR setups actually run — rclone is the pragmatic tool. Define two named S3 remotes, then sync:

# ~/.config/rclone/rclone.conf
# [sourced from https://rclone.org/s3/, NOT EXECUTED IN CI]
[primary]
type = s3
provider = Other
access_key_id = rustfsadmin
secret_access_key = rustfsadmin
endpoint = http://primary-site:9000

[dr]
type = s3
provider = Other
access_key_id = rustfsadmin
secret_access_key = rustfsadmin
endpoint = http://dr-site:9000
Enter fullscreen mode Exit fullscreen mode 
# [sourced from https://rclone.org/commands/rclone_sync/, NOT EXECUTED IN CI]
rclone sync primary:dr-backups-primary dr:dr-backups-dr --progress
Enter fullscreen mode Exit fullscreen mode

Then put that behind cron every 30 minutes. Note rclone sync makes the destination match the source — it deletes objects at the destination that no longer exist at the source. If you want DR to survive an accidental mass-delete on the primary, use rclone copy instead, or enable versioning and Object Lock on the DR bucket.

Step 4: Document & Test the Restore Procedure

Your DR plan is only useful if the person on call can execute it without calling you. I have been that 3 a.m. call; write the runbook for someone who has not seen it before:

DR Runbook: Primary Region Failure

Trigger conditions:

  • Primary region unreachable for > 15 min
  • Major data corruption detected
  • Executive decision to failover

The whole procedure runs about 45–90 minutes. Each step is a separate command block you can paste as-is.

1.Verify DR bucket integrity (5 min)

   aws s3 ls s3://dr-backups-dr/ --recursive | wc -l
   # Compare count to expected object count
Enter fullscreen mode Exit fullscreen mode

2.Promote DR database (15-45 min)

   # Start the instance in the DR region, then stream the dump straight out of S3.
   # NOTE: gunzip cannot read an s3:// URL — you must pipe through `aws s3 cp ... -`.
   LATEST=$(aws s3 ls s3://dr-backups-dr/postgres/ | sort | tail -n 1 | awk '{print $2}')
   aws s3 cp "s3://dr-backups-dr/postgres/${LATEST}db.dump.gz" - \
     | gunzip -c \
     | pg_restore -d production_db
Enter fullscreen mode Exit fullscreen mode

3.Update DNS (2-30 min, depends on TTL)

   # Route53: Change A record to DR instance IP
   # Wait for DNS propagation (monitor with dig + health checks)
Enter fullscreen mode Exit fullscreen mode

4.Verify application health (10 min)

   curl https://myapp.com/healthcheck
   # Expect: {"status":"ok","region":"dr"}
Enter fullscreen mode Exit fullscreen mode

5.Communicate (ongoing)

  • Status page update
  • Internal Slack alert
  • Customer notification (if SLA impacted)

Test this runbook quarterly. An untested runbook is just a theory, and outages are bad at following theories.

What goes wrong with S3 DR?

Replication lag will surprise you

CRR is asynchronous. An object written at T0 might not appear in the DR bucket until T0 + 15 minutes (or longer under load). Your latest backup might not yet be in DR when you need it. This is the failure mode I check first in any DR drill.

What helps: check replication lag metrics before declaring disaster, and keep a force-sync procedure for critical objects.

Encryption keys live in one region

If your primary region uses AWS KMS-managed keys and that region is down... you can't decrypt backups in the DR region unless you've planned for cross-region KMS access. I have seen a perfectly good DR bucket become useless because the key was still in the failed region.

What helps: use client-side encryption (you hold the key) or ensure the KMS key is accessible from the DR region (multi-Region KMS key).

S3 is not a database DR mechanism

S3 is great for DR of things stored in S3. It is not a replacement for database replication. Your PostgreSQL primary still needs streaming replication, logical replication, or Patroni — S3 is the safety net, not the primary mechanism.

TL;DR

  1. S3 is probably already your biggest DR asset — most critical data lands there eventually.
  2. Three patterns: Backup-to-S3 (simplest, RPO=hours), Log shipping (better, RPO=minutes), Active-Active (best, complex).
  3. Start with Pattern 1 (automated backups + CRR) — it's a 1-day setup that covers 80% of DR scenarios.
  4. Test restores quarterly. An untested DR plan is a false sense of security.
  5. S3 complements — doesn't replace — database replication. Use both.

Building a DR target on self-hosted S3? RustFS is Apache 2.0 licensed and its README Feature & Status table lists **Bucket Replication, **Versioning, **Bitrot Protection* and Event Notifications as ✅ Available — the four primitives a DR pipeline actually leans on. Being straight with you: Distributed Mode is still 🚧 Under Testing, so validate your multi-node topology yourself before betting a production DR plan on it. Download here.*

FAQ

Is S3 enough for disaster recovery, or do I need a database replica?

You need both. S3 handles the non-transactional layer — files, logs, backups, artifacts. For PostgreSQL or MySQL you still need streaming or logical replication if you want RPO measured in seconds. I treat S3 DR as the safety net and database replication as the thing that actually keeps me asleep at night.

How long does it take to restore from S3 after a disaster?

You will not know until you time it. Bandwidth to the S3 endpoint, I/O on the restore target, and whether you are doing a full restore or point-in-time recovery all dominate. Run a quarterly drill, record the real number in the runbook, and use that as your RTO. The planning-doc number is fiction.

Should I use AWS S3 or self-hosted S3 for DR?

Both work. AWS S3 with Cross-Region Replication is simpler to configure but costs more (storage plus request and egress fees) and keeps your DR data inside the same vendor blast radius as production. Self-hosted S3 (RustFS, MinIO, Ceph RGW) gives lower ongoing cost, no inter-site egress billing, and vendor independence, at the price of operating the DR storage yourself. Many teams run a hybrid: self-hosted for the primary site, a cloud bucket as the off-site archive tier.

Does S3 Cross-Region Replication protect me from accidental deletion?

Not by itself. CRR faithfully replicates deletions when DeleteMarkerReplication is enabled, so a mass-delete on the primary propagates to DR. Real protection comes from versioning plus Object Lock (WORM retention) on the destination bucket, and from keeping at least one backup copy outside the replication path. Replication is a redundancy mechanism, not a backup.

What is the required IAM setup for put-bucket-replication?

The Role field is a required top-level element of the replication configuration — it is the IAM role S3 assumes on your behalf. That role needs read permissions plus s3:GetReplicationConfiguration on the source bucket, and s3:ReplicateObject, s3:ReplicateDelete and s3:ReplicateTags on the destination. Versioning must be enabled on both buckets before the rule will apply.

Sources

All figures and commands in this article were checked against primary sources on 2026-08-06:

Commands marked NOT EXECUTED IN CI are reproduced verbatim from the linked official documentation but were not run in the environment used to write this article.

Top comments (0)