📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram
Originally published on software-engineer-blog.com.
Two letters apart. Both measured in minutes. Bought with completely different money.
That is why teams argue about disaster recovery for a week without noticing they are talking about two separate things. This article takes the two numbers apart, and then prices both of them against one small shop.
Tuesday, twelve minutes past three
The server that holds every order for an online shop is gone.
The alert fires. Somebody wakes up. By twenty past three they have found the most recent saved copy of the data. That copy was taken at three, on the hour. So they build a new server, and they put that copy onto it. At nineteen minutes past four, the shop can take orders again.
Now count what actually happened.
The shop was shut for sixty-seven minutes. And twelve minutes of orders — the ones taken between three o'clock and the moment it died — do not exist anywhere any more.
Two different things went wrong.
For the next week, the team argued about how to make sure it never happened again. And in that whole week, nobody ever said which of the two numbers they were trying to make smaller. So which one was it? The sixty-seven minutes the shop was shut, or the twelve minutes of orders that vanished?
What this article covers, and what it does not
Three things are covered:
- What the two numbers actually are. One is how long you may be down. The other is how much data you may lose.
- What each one is bought with, because they are not bought with the same money.
- How the standard tiers are priced, and where that price stops being reasonable.
Three things are not covered. This is not how backups work on the inside — there are no restore steps here, and that is a subject for its own article. This is not about which site serves your traffic or how failover flips between two sites, which is covered elsewhere on this site. And this is not one product: no tool and no cloud provider is named anywhere below.
Two dials, not one
Here is the confusion this kills. People hear "disaster recovery" and think there is one dial, and that turning it up makes everything better.
There are two dials. A shop that catches fire is the easiest way to see them.
First question. How long is the shop shut? The fire is out, but the doors still do not open. Every hour shut is customers who go somewhere else. You make that shorter by renting a second shop, already fitted out and ready. And a second shop costs rent every single month, whether or not there is ever a fire.
Second question. How many pages of the order book burned? The shop can open again, but some orders simply do not exist. You lost every order taken since the last time somebody carried a page across the road to the safe. You make that shorter by copying more often. And copying after every single order slows down every single order you ever take.
Now notice how independent those two are. You could reopen in an hour with a whole day of orders missing. You could lose nothing at all and stay shut for a week. Both are possible, because these are two different numbers.
One line of time, two arrows
Draw the life of the failure as a line of time. Mark the moment it broke. Everything to the left already happened. Everything to the right has not happened yet.
Look backwards from the break, to the last moment your data was safely saved somewhere else. That distance is the recovery point objective, or RPO. It is the data you are willing to lose. On Tuesday it was twelve minutes.
Look forwards from the break, to the moment a customer could place an order again. That distance is the recovery time objective, or RTO. It is the time you are willing to be down. On Tuesday it was sixty-seven minutes.
Same instant. Two arrows. They point in opposite directions.
| RPO | RTO | |
|---|---|---|
| Which way it points | backwards, into the data | forwards, into the clock |
| What it measures | work that vanished | time the doors were shut |
| What sets it | how often you copy | how fast you can be serving again |
| On Tuesday | 12 minutes | 67 minutes |
Both are measured in minutes, and that is exactly why people mix them up. Every section below is this same line with the two marks moved.
RPO: the promise is the worst case
Suppose you save a copy of the data every fifteen minutes. What have you actually promised?
Here is the whole thing, in a few lines.
# rpo.py -- what does "we copy every fifteen minutes" really promise?
COPY_EVERY = 15 * 60 # seconds between one saved copy and the next
ORDERS_PER_MIN = 50 # this shop, on an ordinary day
def lost(failed_at, last_copy_at):
return failed_at - last_copy_at # everything in the gap is gone
# The failure does not check your schedule. It lands anywhere in the gap.
worst = lost(failed_at=899, last_copy_at=0) # a second before the next copy
best = lost(failed_at=1, last_copy_at=0) # a second after the last one
print(worst, best) # 899 1 -> the whole gap, or almost nothing
print(worst / 60 * ORDERS_PER_MIN) # 749.2 -> orders that never existed
Everything written between the last saved copy and the failure is gone. That is the definition, and it is the only thing that matters here.
Now, the failure does not check your schedule before it happens. It can land anywhere inside that gap. If it lands one second before the next copy was due, you lose fourteen minutes and fifty-nine seconds of work. If it lands one second after a copy, you lose one second. Both are possible on the same schedule.
And here is the trap. The number you promise the business is the worst case, never the average. A copy every fifteen minutes means fifteen minutes.
This shop takes fifty orders a minute, and an order is worth thirty-eight euros. So the same schedule, said honestly, looks like this:
| How often you copy | Worst case | Orders that never existed | Value |
|---|---|---|---|
| Once a night | 23 h 59 min 59 s | 71,999 | EUR 2,735,968 |
| Every 15 minutes | 14 min 59 s | 749 | EUR 28,468 |
| As each order is taken, about 2 s behind | 2 seconds | 1.67 | EUR 63 |
Look at what changed between those three rows. Not the quality of the copies. Not the storage. Not the restore. The only thing that changed is how often.
That is the only lever RPO has, and every step of it costs you something on the way in. Which is why arguing about backup software rarely moves this number at all.
RTO is a sum, and the restore is the small term
Now the other number, and it has a trap of its own.
Ask a team what their RTO is, and most of them answer with how long the restore takes. Watch what that leaves out. The clock does not start when you start working. It starts when it breaks.
# rto.py -- the clock starts when it breaks, not when you start working.
detect = 4 * 60 # the alert fires, and a human actually sees it
decide = 11 * 60 # somebody decides this is a disaster, not a blip
provision = 9 * 60 # new machines exist, and they are reachable
restore = 26 * 60 # the data is put back onto them
verify = 8 * 60 # somebody checks it is really the right data
cut_over = 9 * 60 # traffic is pointed at the new place
RTO = detect + decide + provision + restore + verify + cut_over
print(RTO // 60) # 67 -> the real number, in minutes
# the one term everybody budgets for, on its own:
print(restore // 60) # 26 -> not even half of the sixty seven
Sixty-seven minutes in total. And the one term everybody budgets for, the restore, is twenty-six of them. That is 38.8 percent. Not even half.
| Term | Minutes | Can money make it shorter? |
|---|---|---|
| Somebody actually sees the alert | 4 | no — people |
| Somebody decides this is a real disaster | 11 | no — people |
| New machines exist and are reachable | 9 | partly |
| The data is put back | 26 | yes |
| Somebody checks it is the right data | 8 | no — process |
| Traffic is pointed at the new place | 9 | partly |
| Total | 67 |
Here is why that split matters. The twenty-six is the part you can buy your way out of, with faster disks and a bigger pipe. The other forty-one minutes are people and process. No hardware makes any of that shorter.
So a team that spends its whole recovery budget on a faster restore cannot get its RTO below forty-one minutes, no matter what else it spends. And it usually does not know that.
If you want the number below forty-one, you have to stop restoring, and start already having a second copy running.
The three tiers, and why you do not choose them
Once both numbers are written down, something useful happens. The strategy stops being a choice.
There are three tiers, and they sit at fixed points on those two numbers.
| Tier | RTO | RPO | What it costs |
|---|---|---|---|
| A saved copy, and a rebuild | hours | however often you copy | storage, and almost nothing else |
| A second site, kept warm | minutes | however far behind the copying runs | a second bill, every month |
| Two live sites, every write waits for both | seconds | near zero | that second bill, plus something on every request |
Cheaper and slower at the top. Faster and far more expensive at the bottom.
For this shop, at fifty orders a minute, the difference between the rows is orders that were never placed at all:
| Tier | Time down | Orders never placed |
|---|---|---|
| A saved copy, and a rebuild | 67 minutes | 3,350 |
| A second site, kept warm | 5 minutes | 250 |
| Two live sites | 20 seconds | 17 |
You do not shop for these three. You write down your two numbers, and exactly one row is left.
What RTO is bought with: capacity that sits there
Moving RTO down means having capacity that is already running when the failure happens. Not capacity you can create quickly. Capacity that is already there.
A second site kept warm is machines that are running, patched, watched and paid for on every ordinary day.
Say the production environment costs four thousand two hundred euros a month. A full warm twin doubles that. Fifty thousand four hundred euros a year.
And here is the shape of that purchase. The bill arrives twelve times a year, every year. The outage it exists for might arrive once. Or never.
That is not an argument against buying it. It is an argument for knowing exactly what you are buying.
What RPO is bought with: waiting, on every write
Now the other mark. This is where it gets interesting, because RPO is not bought with money. It is bought with waiting.
Here is the same order, saved two ways.
# write.py -- the same order, saved two ways. Only the waiting differs.
RTT_MS = 12 # a round trip to the second place, far away
WRITES_PER_ORDER = 6 # one checkout touches six rows
def save_async(row): # RPO is the lag: about two seconds of orders
write_here(row) # the customer waits for this, and nothing else
send_to_other_place(row) # this happens after the customer is gone
def save_sync(row): # RPO is zero: nothing is ever lost
write_here(row)
wait_for_other_place(row) # the customer waits for this too, every time
print(RTT_MS * WRITES_PER_ORDER) # 72 -> milliseconds added to every order
In the first version, you write the order here, and send it to the other place afterwards. The customer waits for the first write and nothing else. RPO is whatever the sending falls behind by. Call it two seconds.
In the second version, you write it here, and then you wait for the other place to say yes before you tell the customer their order went through. Now RPO is zero. Nothing is ever lost.
But look at who is waiting. The customer is. Every time.
If the other place is far enough away that a round trip takes twelve milliseconds, and one checkout touches six rows, that is seventy-two milliseconds added to every order. Forever.
Seventy-two milliseconds does not sound like much. So multiply it out.
| What | Amount |
|---|---|
| Orders a year, at 50 a minute | 26,280,000 |
| Added to each one | 72 ms |
| Added up, per year | 1,892,160 s = 525.6 hours = 21.9 days |
| What it protects | 1.67 orders — EUR 63 — on the one bad day |
Almost twenty-two whole days of human waiting, added up, every single year. To buy back sixty-three euros of orders on a day that may or may not ever come.
That is what driving RPO all the way to zero actually costs.
The step that is worth it, and the step right next to it
Now put the money side by side. This shop loses one thousand nine hundred euros for every minute it is down — fifty orders a minute, thirty-eight euros an order.
Step one. From sixty-seven minutes down to five.
That costs one more full environment: fifty thousand four hundred euros a year. It saves sixty-two minutes of downtime, which is one hundred and seventeen thousand, eight hundred euros per outage.
Break-even is 0.43 outages a year. Have one outage every two years, and it has already paid for itself. That is an easy yes.
Step two. From five minutes down to twenty seconds.
That costs another environment, and now every write waits for the far side. It saves four minutes and forty seconds, which is eight thousand, eight hundred and sixty-seven euros per outage.
Break-even is 5.7 outages a year. And you pay the five hundred and twenty-five hours of customer waiting either way.
| Step one: 67 min → 5 min | Step two: 5 min → 20 s | |
|---|---|---|
| Saved per outage | EUR 117,800 | EUR 8,867 |
| Costs | EUR 50,400 a year | another EUR 50,400 a year |
| Plus | nothing on the request path | 525.6 hours of waiting a year |
| Break-even | 0.43 outages a year | 5.7 outages a year |
| Verdict | an easy yes | almost certainly no |
Same ladder. Same shop. Same arithmetic. The step that is obviously worth it and the step that is obviously not are right next to each other.
This is the shape to remember. Both marks move a long way for money you can put in a budget. It is the last small step — pushing either mark all the way onto the failure itself — where the price stops being money and starts being waiting on every request you will ever serve.
How you actually pick
You ask the business two questions, and you write the answers down before anybody says the word "backup".
- How long may we be shut before it really hurts?
- How much work may vanish before it really hurts?
Those two answers are your RTO and your RPO. Everything after that is arithmetic, and the arithmetic is genuinely this small.
# pick.py -- you do not pick a strategy. You pick two numbers.
# Ask the business first, and write both answers down before anything else.
rto_minutes = 5 # how long may we be shut before it really hurts?
rpo_seconds = 2 # how much work may vanish before it really hurts?
def tier(rto_minutes, rpo_seconds):
if rto_minutes > 60 or rpo_seconds > 900:
return "a saved copy, and a rebuild" # cheapest. Hours.
if rto_minutes > 1 or rpo_seconds > 0:
return "a second site, kept warm" # minutes
return "two live sites, and every write waits" # seconds, and it hurts
print(tier(rto_minutes, rpo_seconds)) # a second site, kept warm
The two numbers came first. The tier came out of them.
Three questions you can answer this afternoon
First. If the main copy of your data disappeared right now, how many minutes of work would be gone? Not on average. In the worst case. If nobody can say the number, you do not have an RPO — you have a schedule.
Second. Add up your six terms: seeing the alert, deciding it is real, getting machines, putting the data back, checking it, switching traffic over. What is your floor once you remove the restore? If that floor is already above what you promised, buying a faster restore cannot save you.
Third. Who agreed to these two numbers? If the answer is "the engineers", they are estimates. They become objectives when the business says them out loud.
The verdict
| RTO | RPO | |
|---|---|---|
| The question | How long may we be down? | How much work may vanish? |
| Measured | forwards from the break | backwards from the break |
| Set by | a sum of six terms, only one of which is the restore | how often you copy — nothing else |
| Bought with | capacity running on every ordinary day | a wait added to every write, forever |
| You pay | whether or not anything ever breaks | on every request you will ever serve |
| Fails quietly when | the budget all goes to a faster restore | the schedule is quoted as an average |
RTO is how long you may be down, and you buy it with machines that sit there costing money while nothing is wrong.
RPO is how much work may vanish, and you buy it with a wait added to every write, forever.
Write both numbers down before anybody says the words "backup", "standby" or "second region". After that, the tier is the only thing left to read off.
You do not pick a disaster recovery strategy. You pick two numbers, and the strategy falls out of them.
References and further reading
On the two definitions
- NIST Special Publication 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems (NIST, 2010) — the formal definitions both terms come from: RTO as the maximum time a resource may be unavailable before the impact is unacceptable, and RPO as the point in time to which data can be recovered, which in practice is how much data loss the organisation can tolerate: nvlpubs.nist.gov
On writing the two numbers down with the business
- Laine Campbell and Charity Majors, Database Reliability Engineering (O'Reilly, 2017), ch. 2 "Service-Level Management" and ch. 7 "Backup and Recovery" — the same two numbers, under the names availability and durability, treated as service-level indicators agreed in advance. Durability is defined there in exactly this shape: "in the event of a system failure, no more than the past two seconds of data can be lost." Chapter 7 then opens the recovery-strategy discussion by sending you back to those objectives first.
- Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara and Stephen Thorne (eds.), The Site Reliability Workbook (O'Reilly, 2018), ch. 2 "Implementing SLOs" — how a reliability target is chosen with the people who own the product rather than by the team that will engineer to it, and why the rationale behind the number gets written down with it: sre.google/workbook/implementing-slos
On the restore being the small term
- Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering (O'Reilly, 2016), ch. 26 "Data Integrity: What You Read Is What You Wrote" — "No one really wants to make backups; what people really want are restores", and the requirement that recovery finish "well within the uptime needs of a service", which is the RTO by another name: sre.google/sre-book/data-integrity
On what the last step of RPO costs
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), ch. 5 "Replication", section "Synchronous Versus Asynchronous Replication" — waiting for the second copy to acknowledge is what removes the loss window, and it is also what puts an unbounded wait on the write path, which is why almost all replication is asynchronous in practice.
On the last rung costing more than the outage
- Betsy Beyer et al. (eds.), Site Reliability Engineering (O'Reilly, 2016), ch. 3 "Embracing Risk" — reliability is not maximised, it is targeted: "an incremental improvement in reliability may cost 100x more than the previous increment", and the target is treated as both a minimum and a maximum: sre.google/sre-book/embracing-risk
If a reference you would expect is missing, say so in the comments and I will add it.
Watch the full episode: RTO vs RPO on YouTube
Top comments (0)