When building applications, handling concurrency issues is a critical aspect of ensuring data integrity. MongoDB lacks an API for conditional expressions in inserts, a feature that is standard in databases such as DynamoDB, making dealing with concurrency harder. How can we effectively manage situations where multiple entities attempt to write to the database simultaneously, potentially compromising data integrity? In this article, we will explore three strategies to address this challenge: FIFO Queues, Distributed Locks, and our main focus, Conditional Put.
Concurrency issues occur when multiple entities attempt to write to the database simultaneously. This can lead to conflicting writes, potentially compromising data integrity. Guaranteeing that only a single entity will be able to write on a database given a condition, is crucial for some systems.
To address this issue, there are a few strategies we can use. In this post, we will go through three of them: FIFO Queues, Distributed Locks, and our main focus, Conditional Put.
Before diving into possible solutions, we need to identify our problem. Our scenario is the following: We're building an application where Users can request Races which can be accepted by Drivers. We will focus on the User requesting Races part. Our goal is that a User shouldn't be able to request two Races at the same time.
Note that this is an extremely simplified version of the data modeling. The focus of this post isn't the data modeling but how to solve a problem like the one presented.
Now that we have our 'schema' and problem, let's talk about how we can approach to solve it.
FIFO Queues
FIFO stands for 'First In First Out'. In our case, it would be a message queue without any concurrency, guaranteeing that all the requested writes would be always tried in the order they were requested. We can just add a check if the item already exists. In a way, we can say that FIFO Queues would guarantee that there wouldn't be any concurrent attempts at writing on our database.
The problem with this solution is scaling. Imagine we have thousands of operations being realized at the same time; a single queue wouldn't be very fast to handle it. This could result in a poor user experience due to latency. The advantage is that it is very simple to implement, especially if you already have a message queue system implemented in your application. All you would need to do is create a new queue for it.
Distributed Lock
Distributed Locks would be a more robust solution to our problem. In a Distributed Lock system, the system distributes tokens to consumers. We add a lock to our process and check if the consumer has a valid token. If not, we just return an error. This would allow us to still run concurrent processes; we could have only a single valid token available at a time for a User, per race requested. Whichever request acquires the lock first, wins the token and can write to the database.
This is a pretty robust solution, and it would also scale nicely. The problem with Distributed Lock systems is that they are very complex. Going deeper into them would be a subject for another article, or possibly even several more. Another point to consider is that you most likely would be introducing a new layer to your application, maybe something like Redis or possibly a more robust locking system.
Conditional Put
This is the main focus of this post, and probably why you're here. While MongoDB has a conditional expression system for updates with the $cond operator, it doesn't have one for inserts, which is where our problem lies. But that doesn't mean it's not possible to make Conditional Puts in it; we just need to be a bit more creative.
We can solve our problem by adding a new field to our Race model, which we will call conditionId. This field will be an ObjectId with a unique constraint attached to it.
So what is this id? What is it going to reference? This is where we get creative; this id will be the id of the last race requested by the user. Due to the unique constraint we added to the field, we can guarantee that only a single document will exist with it; all other attempts will fail at the database level.
Here's some pseudo code of how our logic would look:
const raceConditionIdGet = async (user: User) => {
const [lastRace] = await Race.find({ user }).sort({
_id: -1,
});
return lastRace._id;
}
const raceCreate = async ({ user, ...args }) => {
const conditionId = await raceConditionIdGet(user)
const race = await new Race({ ...args, user, conditionId }).save()
if(!race) {
return error
}
return race
}
Now you may think, what if this is the User's first Race? There won't be a valid conditionId, so what do we use? In this case, we can use the User's own id.
const raceConditionIdGet = async (user: User) => {
const [lastRace] = await Race.find({ user }).sort({
_id: -1,
});
if(!lastRace) {
return user._id
}
return lastRace._id;
}
You can see that our conditionId acts very similar to the idea of the Distributed Lock token; the advantage is that we didn't introduce any new system to our codebase. But just like any other solution, it has trade-offs. While storage is cheap, does it make sense to save this id? What if we have multiple conditions? Would it be a good idea to just keep saving new ids, maybe even repeated ones in the same document?
Conclusion
Just like pretty much any other problem like this one, there is not actually a right answer; it all depends on your context. If you already have a message queue system implemented and are not expecting a lot of demand, the queue system will work just fine for you. You may (probably) be building something way more complex than our contained example, so the distributed lock may make more sense for you.
If you want to work in a startup in its early stages, This is your chance. Apply today!
Woovi is a Startup that enables shoppers to pay as they please. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.
If you want to work with us, we are hiring!
Top comments (2)
How about setting {optimisticConcurrency: true} on the schema?
ref: mongoosejs.com/docs/guide.html#opt...
The way optimisticConcurrency option works, is that it tracks the versioning of a document. It works for update scenarios, but doesn't solve the problem in insertions.