The proof of concept took an afternoon. Somebody created an account, accepted whatever region was preselected, deployed a couple of services and showed it to a stakeholder who liked it. Eighteen months later that region held our production database, our object storage, our queues and about four years of regulatory retention, and nobody had ever made a decision about it.
The consequences arrived one at a time, each small enough to absorb. Most of our users were on the other side of an ocean, which added a bit under a hundred and forty milliseconds to every round trip, and because the application was chatty rather than slow, that turned into a page load people described as sluggish without ever filing a ticket. Then legal asked, during an unrelated review, where customer records physically lived, and the honest answer involved explaining that nobody had chosen. Then a managed service we wanted was not offered there, so we ran an older self-managed version instead and inherited the operational work we had adopted the platform to avoid. Then the analytics team stood up their warehouse somewhere closer to head office, and we began paying to move data across regions every night, forever.
What makes region selection nastier than it looks is that it is one of the few genuinely sticky choices in an environment that markets itself as elastic. Compute moves easily. Stateful services move with an outage, a migration project and a real risk of data loss. By the time anyone is senior enough to care about the answer, changing it costs more than living with it, so we live with it, and we call that architecture.
I now treat a small number of choices as one-way doors and insist they get five minutes of deliberate thought even during a prototype: region and data residency, account and subscription boundaries, the identity provider, and the primary datastore. Not a design document, just a sentence recorded somewhere about why, so the next person inherits a decision instead of an accident.
The dangerous defaults are not the ones that are wrong. They are the ones that are fine, quietly become permanent, and then have to be defended by people who were not in the room.
– Serguey Shinder
Top comments (0)