Of Software and Systems : Scaling and it's costs

Moving from a single-server application to a distributed system feels a lot like moving out of a studio apartment into a crowded house. Suddenly, you aren't the only one using the kitchen, and if you don't start coordinating, someone is going to put their groceries right on top of yours.

When your C# or Python application is running on just one instance, things like database writes, memory caches, and background jobs are highly predictable. But the moment you scale horizontally to two, three, or fifty instances, your application memory starts lying to you. What one instance thinks is true has already been changed by another. If you aren't prepared, you will quickly run into race conditions, duplicated payments, and out-of-order data corruption.

Here are the four most common ways your application will break when scaling out, and exactly how to fix them in both the .NET and Python ecosystems.

When scaling to multiple applications, we need to watch out for potentially four things that could break:

1. Racing to write

Imagine we have three instances of the same arbitrary application, and all three are connected to the same database for their write operations. If somehow, by some common chance, more than one instance attempts the same write operation, it could have a double/triple effect on that record/row in the database. An example of this could be a simple bank, say:

C#

var user = await db.Users.SingleAsync(u => u.Id == id);
user.Credits -= amount;              // decided from a snapshot
await db.SaveChangesAsync();         // written much later

or Python

from sqlalchemy import select

# Fetches the snapshot into memory
user = await session.scalar(select(User).where(User.id == id))
user.credits -= amount 

# Writes the snapshot back, potentially overwriting other changes
await session.commit()

How do I fix it?

There are three strategies to tackle this pickle, one way is to ensure that both the read and the write is counted as one atomic operation, another is "knowing" when a snapshot has become stale, or lastly a "uniqueness constraint" that forces only one operation to succeed out of the multiple.

One Atomic operation

In the .NET world, the Entity Framework core (from version 7 onwards) library provides a nifty ExecuteUpdate method that allows you to "save data to the database without using EF's traditional change tracking and SaveChanges() method" as per the Entity Framework Core documentation on ExecuteUpdate , it skips the whole shabang that we're used to with Entity framework, just one single relational SQL UPDATE statement that is automatically respected in the SQL world and locks the row it's looking at.

In python's Django, that would be the F() expression, combined with the .update(), and it does the same thing as in the .NET world in skipping all of the formalities like fetching the object into memory, modifying it and saving it back to the DB. The F() expression is powerful because it refers to the current value of the field in the database, and the update performs the statement directly in the database.

This is because it has to use a locking mechanism to stop other instances from accessing the database, the core assumption of this approach is that these things are likely to happen, and therefore we need to be extra safe around the access to the DB.

Snapshot staleness

This is something a little more optimistic, in that anticipates that this sort of issue does not happen often. And even if it does, it can ensure uniqueness by providing a unique key or timestamp, and when rejected it can potentially afford to throw an error to let the user or consumer know that "hey, something changed between your read and your write".

Using Uniqueness to our advantage

Modern databases give us the ability to index rows by whatever we want, meaning we can make a certain column(s) a point of reference for when we insert. So imagine an operation that aims to insert an account with an email, but in scaling, the operation multiplies somehow. In that case, we use the email as a unique constraint/index that the database will automatically take care of for us. So it will reject an insert operation if the indexed value already exists.

2. Duplication effect

This is when you initiate a "unit" of work twice, whether by accident or intentional, and the side effect of that unit of work happens twice. Sounds dangerous in some industries, imagine banking 🫣 e.g withdraw 1000 twice or 5 times. It's funny because every mainstream broker gives us a guarantee of an at least once delivery, but guarantee an exactly once delivery.

But why is that? Well, Microsoft has an opinion on that question itself, they say that "It's impractical to guarantee exactly-once delivery across a distributed system" and that "The durable solution isn't to eliminate duplicate delivery but to make the consumer process it correctly". So it's our responsibility to make sure we handle things on our end correctly, a broker cannot/will not be responsible for all of it's consumers. I can't open a restaurant and expect to monitor how everyone eats, making sure they don't leave crumbs on the table etc, at the restaurant and at their homes lol

How do I fix it?

Since it is our responsibility to make sure that the consumer handles the messages correctly and "exactly once", we need to use the "Idempotent Consumer" architecture pattern. It covers three area/fixes that work in tandem to create this fix.

Having a stable Deduplication key

"Deduplication" is the act of removing multiple identical entities from a given process, list or any type of container - this my own attempt at sounding smart. So in this case, we want to ensure that the message being sent has been accounted for by giving it a "key" of some sort. We can't rely on the key that lives in the broker itself, because it might malfunction and send two of the same. We, again, need to make sure that sits with the application.

A "business identifier" is something that we would refer to in this case. This is a sort of identification that only the business knows about and can reference when needed during querying performance, bug investigations, operations etc. An example is Order 12345, if the sender sends that, and our consumer application knows that each order being sent is unique and uses that for deduplication, then it doesn't matter how many times the broker messes up and sends multiple messages; it will not be consumed again because that order has already been accounted for in the first run.

Inbox pattern

We can combine the act of deduplication with keeping a record of what has been processed, the place where we keep this information is called the "inbox", hence the term "inbox pattern". So the process is:

1. Event/request comes in
2. It has a unique business identifier like OrderId 12345
3. The consuming/receiving application uses this id to check it's inbox to see if it has processed it
4. If not it will then process it and in the same transaction it will record the BID to the inbox
5. If done so already, then it will be dealt with how the business sees fit, whether throwing an error, providing a 200/202 OK message

These two approaches MUST happen at the exact same transaction

These two actions (dedupe key and recording in the inbox) must both be within the same unit of execution. If not, then this pattern doesn't solve the duplication issue at all. Either both are successful or both fail, this is what makes sure that processes are protected from duplicate procedures.

Natural idempotency

We can also rely on the business logic itself to save us from duplication. One way is through the use of naturally idempotent processes, where it produces the same end state with every request it makes. An example is calling an endpoint to change name = "Harvey", name will always be Harvey no matter how many times it gets called.

3. Local state

One of the things that small applications usually do is create things/jobs/processes that only the one process (the app) can acknowledge and use. Things like Semaphore locks, caches, in memory counters and sticky sessions etc. This is fine if it was one instance, but what happens when we make two instances, worse yet, three or four?

The applications in those cases, now, "think" that only they have a lock on the access to the database, or their in-memory cache is the only one being created to keep track of whatever it is, but we both know that they're wrong when looking at the wider picture. This is because they have now potentially created multiple processors of the same request, whether by event or by standard request, and no amount of local lock, in-memory cache and the rest of those local fools can stop the unintended side-effects.

How do I fix it?

Luckily for us, we know the problem, too many of these things think on a local level, meaning they only care about their machine instance. We need them to care more about the wider picture, which has more instances of the same application. There are three main ways to deal with this:

"I'll let you have it for a bit and then I'll give it to someone else" (otherwise known as Leasing)

We know that locking, for instance, will require a sort of coordination between the other instances to let them know "hey, I'm using this now, so wait your turn until I'm finished or I'm dead". Redis is one of those popular technologies that helps with this sort of thing, it helps do the heavy lifting of coordination between your instances so an application doesn't have to. So before executing a database related action, the first app will check with the coordinator (the Redis instance, for example) and ask if there is any other of it's instances using the lock at this time. If not, it will claim it for a certain time period (using time to live settings TTL settings) and tell it to block any other instance from using it (otherwise known as Distributed locks).

Leader Election

While leasing is based on keeping the resource until you finished using it or time expires, leader election is based on how to get to that stage to begin with. In order to get the privilege to lease, the candidates (competing servers) must vote for who should get the resource. They vote for whichever candidate requests to get the lease first. So if server 2 requests it first, it will vote for itself, and because the other two servers didn't get there first, they will vote for it too, and then a leader would be effectively elected from that.

Being able to choose a leader for a specific action or task sort of removes the single point of failure when it comes to leasing alone. With leasing alone, you're essentially going to a bulletin board and asking "is this resource free", if that bulletin board is offline, any and every server relying on it will now not be able to access the DB, because it is designed to do so. It doesn't know about other servers, and that is what leader election fixes as a side effect, it allows servers to talk to each other, and doesn't need a bulletin board to be active. It allows sending messages to each other when it comes time to vote for a leader.

The senior option, try and avoid this complexity

Simple as that, trying to make sure you steer clear of complexity like locking resources is a daily, line by line, struggle. It also applies to this situation, how do we do it, you say? Well, for us to solve this problem of duplication through local state fallacy, we'll need to try and design a system that encourages natural idempotency. A request that targets an end state of an entity is different than a request that can add to that state indefinitely. I.e. if I said "add 10,000 to my account", it's different than saying "My account should now have 20,000". One request can potentially cause an issue during duplication, and one just does the same thing no matter how many times it is requested.

4. Broken Ordering

Events consumed by multiple consumers can sometimes get into a tricky situation of consuming events out of the order they were supposed to be consumed. This is what is called broken ordering, and annoying scenario where order_placed comes after order_cancelled, meaning the customer may think that their order has been cancelled, but it's still going! I think it's most commonly seen in event driven architecture, don't quote me on it.

How do I fix it?

There's a fix for everything except for death and taxes, but they're working on it! In this scenario, we're coming across a problem of un-ordered events being consumed, we need to make sure (at the broker or consumer) that some how they are ordered, or at least made sure of that it is ordered.

"For this order_id_1234, I will be the only consumer for it..." (Partitioning)

And it continues to say:

"no matter which events for it come up and how many"

Keep that in your mind for later, but for now this is a great solution that allows a single consumer to be able to be given something in the exact same order as they are published. Azure service bus provides this feature using the SessionId property in it's ServiceBusMessage class. Small caveat, it's not on by default, it needs to be turned on/ enabled during the queue creation process, there will be a property in the UI wizard or which ever CLI you use for the enable-session property. I believe Kafka (in Python world here) uses "Partitions" when sending a message by providing a key argument in it's .send method. This allows kafka to create a has for that key, which will always be associated with that particular partition/consumer.

This seems like a good idea, because it doesn't need to create any extra logic, having the infrastructure do all the work to get the event message to the right place. And it could acheive a great ordering mechanism as a result. However, with this, it could mean that the consumer could do all the work for a particular event, while the other consumers lie on their butts and do nothing because "hey, that orderId is not associated with us". When I mean all the work, I mean when 50000 requests come in and it deals with the same sort of key, which means now that the consumer is going to be backlogged to hell, because it's the only one responsible.

Event Versioning (stale messages get rejected)

Instead of relying on the infrastructure to funnel the messages to the right place, we can just let them fall anywhere, and we would deal with the ordering at the database level. It's a little polar opposite. stamping a version on the events lets the business logic understand what part of the lifecycle this event belongs to, and they can choose to do with that information what it wants. And then it gets to the database level, where it judges that event's version, to see if it is deemed an "old" event, by which it could reject it and say "no need, we're already at this stage/version, your existence is obsolete".

The advantage of this is obviously the throughput. any consumer can pick any event version up and consume it how they like. So when that 50000 orders come through with different information and different versions, the throughput can sky rocket and it can deal with throughput well. Buttt, we now have to do the leg work of sorting the versions out and their logic, which also means not only deeming if something is "old", we also need to know if something is "too far ahead". For example, we can't have shipped V2 event before an ordered V1 event. What can be done about this is to make a, small sort of holding, these events so that the system can keep track of what it has and what it hasn't from the event lifecycle, for that particular entity it is enquiring about.

This also leads to another issue which goes back to contention for the database access, a bunch of well to do processes thinking the same thing still doesn't mean that many of them won't be blocked out for two seconds while other processes are finishing access the DB. This can create huge problems for the 50k example, where there would be a large backlog just to attempt to update the database, not even the update itself.

Conclusion

Scaling an application isn't just about spinning up more servers or containers; it is a fundamental shift in how you trust your data. Once you cross the threshold into distributed systems, you have to assume that networks will drop, brokers will double-deliver, and multiple instances will constantly trip over each other trying to access the same database row.

The underlying theme across all of these fixes is pushing the responsibility of correctness to the right layer. Sometimes that means letting Azure Service Bus handle the ordering, sometimes it means letting the database handle the row-level math, and almost always, it means designing your application logic to be naturally idempotent. By anticipating these four bottlenecks and applying the right patterns to your .NET or Python code, you can ensure your system remains predictable no matter how large it grows.

Thank you for reading, hope you enjoy this latest learning entry. Until next time! Peace, love and Code 🚀