re: Scalable architecture without magic (and how to build it if you’re not Google) VIEW POST

VIEW FULL DISCUSSION
 

With a graph database like Riak, your capacity is no longer limited. When you’re running low, you just buy a new storage server and add it to the mix.

Oh, if only that were true...

When you add a new server to a storage backend, you have to migrate data to it. That's a major load on the servers and the network. Do you have enough capacity? Also, will your partitioning/sharding scheme scale? You can only migrate data to a new server if your partitioning scheme is granular enough. It's easy to find scenarios where you have no granularity left to partition.

Any database product that says, "You just add servers and it takes care of itself!" it engaging in false advertising. Certain parts of the scaling operation may be built in, but the operational headaches and risks are still there.

It's easy to get caught up talking about horizontally scaling stateless programs, but it's not actually that interesting, and a team that effortlessly scaled a stateless program did so because a specialized team was taking care of the scaling problems in the stateful systems they were using and trying to abstract as much as possible away. You can only get so far, though. For example, many developers who work in stateless environments will see a deadlock exception from MySQL and immediately say, "Oh, something's wrong with the database" and forward it to the infrastructure team. But that's not infrastructure. That's concurrency in the application logic, that is, state.

 

When using a distributed system, of course you should go for atomic data model. Of course if you just store the data as is, it wouldn't distribute.

code of conduct - report abuse