DEV Community: JingIsCoding

Efficiently Manage Distributed Locks with Redis: A Go-Based Solution

JingIsCoding — Sun, 20 Oct 2024 20:14:00 +0000

Distributed locks are essential in systems where multiple processes compete for shared resources. Whether it’s database access or file modifications, preventing race conditions is crucial. In this article, I will propose a Redis-based distributed locking implementation in Go, which can be used to synchronize tasks across multiple servers.

The main challenge in distributed locking is ensuring that locks are released in case of failure, avoiding deadlocks, and managing contention. Our Redis lock library, built in Go, solves these issues by ensuring that locks are automatically released and queued up requests are managed efficiently.

This library is built with several features designed to make distributed locking simple and reliable:

Automatic Lock Expiration: Locks are automatically released after a timeout if they are not explicitly unlocked, preventing deadlocks.
Queueing Mechanism: Contending requests are queued, ensuring that they are granted access in a first-come, first-served manner.
Event Subscription: The library leverages Redis’s Pub/Sub mechanism to listen for key expiration or deletion events, ensuring lock handover is efficient. Now, Let’s start by diving into the components and understand how they work from a high level:

The Role of LockManager

The LockManager plays a key role in managing the lifecycle of locks, handling the communication with Redis, and coordinating the locking requests. It’s responsible for:

Acquiring locks: It handles requests to acquire locks on specific keys in Redis, ensuring that only one process or thread can hold a lock on a given key at any time.
Queueing lock requests: If a lock is already held by another process, the LockManager adds the lock request to a queue and waits for Redis notifications indicating that the lock has been released or expired.
Releasing locks: It ensures that locks are released correctly by verifying that the process attempting to release the lock is the one that holds it (based on the unique lock value).
Listening to keyspace events: The manager subscribes to Redis keyspace events, such as key expiration or deletion, to know when locks are released and to notify waiting processes.
Managing multiple locks: The LockManager can handle multiple lock requests simultaneously, making it suitable for distributed systems with many concurrent processes. The Lock function of LockManager takes a context and a key, attempts to acquire the lock, queues requests that cannot immediately obtain the lock, and it will return a Lock struct that we we will talk about in later sections.

func (manager *lockManager) Lock(c context.Context, key string, ttl time.Duration) Lock {
    ...
}

The Lock function is designed to:

Generate a unique value for each lock attempt.
Ensure that only the process/thread that acquires the lock can release it.
Use Redis’s atomic operations to safely acquire the lock.
Queue lock requests and listen for key expiration events via Redis Pub/Sub.

The Lock object

Now, once you obtain the Lock object, you have access to Unlock and Wait functions are designed to work within the object. These functions are critical for managing the lifecycle of a lock and handling the result of acquiring it.

Unlock Function
The Unlock function is responsible for releasing the lock when the thread or process is done with the resource. It ensures that only the thread or process that owns the lock (i.e., the one that holds the correct lock value) can release it. Let's break down how this works:

func (lock *Lock) Unlock() error {
    return lock.manager.releaseLock(lock.key, lock.value)
}

Wait Function
The Wait function allows the caller to wait until the result of attempting to acquire the lock is available. This is particularly useful in cases where lock contention occurs, and a process is queued, waiting for the lock to become available.

func (lock *Lock) Wait() result {
    return <-lock.resultChan
}

Explanation:
Channel-based Waiting: The Lock object has a resultChan channel, which is used to communicate the result of the lock acquisition attempt. When the lock is acquired (or failed), the result is sent through this channel. The Wait function simply blocks until a result is available.

Non-blocking Execution: When a process attempts to acquire a lock using Lock(), it doesn’t need to block the entire thread while waiting. Instead, it can call Wait(), which will block only until a result is ready. The resultChan allows for asynchronous communication between the locking logic and the calling code, making the design non-blocking.

Result Object: The function returns a result object:

type result struct {
    Ok    bool
    Error error
}

In summary, the key features of this library is its ability to handle high concurrency while ensuring that locks are released in a timely manner. By using Redis’s TTL feature, locks are automatically released if the process holding the lock fails.

Redis-based distributed locks are a powerful solution for managing shared resources in distributed systems. This Go library makes it easy to implement robust locking mechanisms that are scalable, efficient, and fault-tolerant. Check out the repository here and start building reliable distributed systems today!

Interested in contributing or have questions? Feel free to open issues or pull requests on the GitHub repository.

What a number

JingIsCoding — Wed, 20 Dec 2023 17:32:44 +0000

Here at chowbus.com, we are building Point of Sales solution for restaurant, one of the most essential part of the application logic to accurately calculate subtotal, tax, discounts, fees and totals for an order, While this might seem like a straightforward task initially, it entails some nuances.

As a standard practice, we employ big integers to represent monetary amounts. This helps us avoid floating-point precision errors that can occur during addition, subtraction, and multiplication between integers. Concretely, $100.00 is represented as the integer 10000 in our system.

Discount in our user case comes with either fixed amount or percentage form, and they can be applied on either order or dishes, Adding complexity, we must distribute the order discount proportionally among each dish when customers opt to pay separately for different items. The allocation of the discount respects the subtotal of each dish.

Consider the scenario where there is an order discount of $1.00 with two dishes priced at $4 and $6. In this case, dish 1 would carry a discount of $0.4, and dish 2 would carry a discount of $0.6. Handling cases where the discount is not evenly divisible by the dish subtotals requires us to distribute the remainders as evenly as possible. For instance, if the discount is $1 with three dishes priced at $3 each, each dish would carry a discount of $0.33, with a remainder of $0.01. This remainder is allocated to one of the dishes.

To demonstrate the point, let’s look at some code.

type DiscountType string

const (
 FixedAmount DiscountType = "fixed_amount"
 Percentage  DiscountType = "percentage"
)

type Discount struct {
  // other fields
  Amount int64 // precision to 4 decimal places, divide by 10000 when used as precentage
  DiscountType DiscountType
}

func (discount *Discount) Apply(subtotal int64) int64 {
  if (discount.DiscountType == FixedAmount) {
      // no easy way to get max or min, unless convert to float64 and use math
      if discount.Amount > subtotal {
        return subtotal
      }
      return discount.Amount
  } else {
      // have to switch between number types
      return int64(float64(subtotal) * float64(discount.Amount) / 10000)
  }
}

And the code to allocate order discount to dishes.

var allocatedDiscount int64
for index, dish := range order.Dishes {
  weight := float64(dish.SubTotal) / float64(order.Subtotal())
  discount := int64(math.Floor(float64(orderDiscount.Amount) * weight))
  dishDeductible := dishItem.SubTotal - dishItem.PreTaxLineItemsDiscount - dishItem.PreTaxOrderDiscount
  // make sure each dish can not be deducted to negative subtotal
  if discount > dishDeductible {
    discount = dishDeductible
  }
  if discount > 0 {
    allocatedDiscount += discount
    order.Dishes[index].PreTaxOrderDiscount += discount
  }
}

// allocate the remainder
remainder := orderDiscount.Amount - allocatedDiscount
dishSize = len(order.Dishes)
for i := 0, i < remainder; i++ {
  dishIndex := i % int64(dishSize)
  item := cart.CartData.DishItems[dishIndex]
  if item.PreTaxOrderDiscount >= 0 && item.SubTotal-item.PreTaxLineItemsDiscount-item.PreTaxOrderDiscount > 0 {
    cart.CartData.DishItems[dishIndex].PreTaxOrderDiscount += 1
  }
}

As we can see, there are lots of type casting between numbers, and what is more unpredictable in this code is that, if the order subtotal is 0, then weight is +/-Inf, and we learnt it the hard way.

Admittedly, it is true that here is ample room for code refactoring to enhance cleanliness, However, the points I am trying make is that we have to deal with this kind of logic through out the calculation logic, which makes the code tedious to write and hard to read for following reasons.

Have to constantly type casting between number types.
Need to shift the decimal digit for example from 10 to 0.01or vice versa.
Need a clear way to round up or down.
Need to make sure division is safely handle regardless the number types.
That is why we open source this library https://github.com/JingIsCoding/number from our code base to mitigate some of the issues. so we can make the code looks more like:

type DiscountType string

const (
 FixedAmount DiscountType = "fixed_amount"
 Percentage  DiscountType = "percentage"
)

type Discount struct {
  // other fields
  Amount number.Number
  DiscountType DiscountType
}

func (discount *Discount) Apply(subtotal int64) number.Number {
  if (discount.DiscountType == FixedAmount) {
      return discount.Amount.Min(subtotal)
  } else {
      return discount.Amount.ShiftDecimal(-4).Multiply(subtotal)
  }
}

var allocatedDiscount number.Number
for index, dish := range order.Dishes {
  weight, err := number.Of(dish.SubTotal).Divide(order.Subtotal())
  // deal with divide by 0
  if err != nil {
    return
  }
  discount := weight.Multiply(orderDiscount.Amount).RoundDown()
  dishDeductible := dishItem.SubTotal - dishItem.PreTaxLineItemsDiscount - dishItem.PreTaxOrderDiscount
  // make sure each dish can not be deducted to negative subtotal
  discount = discount.Min(dishDeductible)
  if discount.IsGreaterThan(0) {
    allocatedDiscount = allocatedDiscount.Add(discount)
    order.Dishes[index].PreTaxOrderDiscount += discount.GetInt()
  }
}

Hopefully you would find this library useful, and leave feedback to the repo if you like.

How we build and maintain a Golang API service

JingIsCoding — Wed, 27 Oct 2021 19:29:39 +0000

Building a scalable, secure, high available api service is not an easy task, especially if there is rapid development and deployment been constantly made. It requires clean and tests-covered codebase, robust infrastructure, real-time visibility and monitoring into system behavior and performance, error reporting and incident triggering system, and more importantly, people need to foster an engineering culture that aims at digging out the root cause instead of putting out patches to fix the immediate problem, beware of the tech debt and understand the tradeoff of short-term deliverables versus long term maintainability.

Here at chowbus, we have been building microservices on Golang/Ruby for a few years now, we are still learning and experimenting in various areas. Meanwhile, there are also some working practices/paradigms I’d like to discuss with the community. In this article we will share a boilerplate that is a simplified version extracted from the production codebase and we will talk about some basic ideas that should help maintaining the project quality and extends its lifespan.

Please refer to https://github.com/JingIsCoding/api-server-boilerplate for a quick setup. We will not dive into the details of the boilerplate in the article as the readme file provides a much deeper introductions on different aspects of the codebase.

Hierarchy and data-flow

One general rule when build an application is to keep modules/layers loosely coupled with each other. In the context of Golang, you might want to always declare interfaces between modules/layers, that will make tasks easier like writing unit-testings since you could easily mock out dependencies to cover all logic path.

Another aspect is to be careful about the object that you pass down the call stack, think about if you pass a request object from client all the way to the data layer. This will implicitly bind all of the components together and make it hard to reuse the underlying logic if you ever need to add another endpoint.

Once the application is up and running in production and clients or other services start to consume the APIs, it then comes the real challenge where we need to alter the existing behavior and provide more functionalities to meet new business needs but have to not break existing clients that might be running on different versions. In the following sections we will talk about some general ideas to keep your application healthy.

Clean code

If you have not, I would strongly recommend you take the time and read this book https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882.

There are many good things in this book, one particular thing can not be emphasized enough is paying attention to the naming of the variables, function and modules, and keep revisiting if you could think of something better. Just mention a few cases here.

Be specific as much as possible.

instead of use variable name like item or data

for _, item := range productService.GetProducts() {
   ...
}

Better to just name thing as it is

for _, product := range productService.GetProducts() {
   ...
}

that way, you will know right away what you are dealing with and refer to it as product in the block.

Single(Limited) responsibility

With the growth of complexity of the business logic and expansion of functionalities, unavoidably, we will find both the length of the source file and the length of certain functions get longer and messier to a point where one file might be 1000 lines long and one function that takes 10 arguments with nested if else blocks, which later becomes untestable because the tests cases will grow exponentially with the number of arguments. To manage this better, alway try to ask the questions that:

Does the function do its name implies? For example, there might be a function called createUser and in the code it not only creates the user but also sends out an welcome email. If that is the case then we should move the sending email to another function and test them separately, if they have to be called in one transaction, then we could wrap them into another caller function called createUserAndSendEmail so other ppl will know the side effects of the function on this abstraction level.
Should we break up one file into smaller (more specific) files? Assuming we have a order.go file that deals with all order related functions, when it starts to get out of hand we could try to break it into files that concerns different aspects of this domain. For instance, we could create a order_creator.go that specifically deals with order creation procedures, and a order_history.go that keeps the history for user to query at anytime, and maybe an order_report.go file that is for statistics purposes.

Protect the data models

Models are at the core of your application, and remember you will not be able to change your data model easily because you always have to carry the historical data for compatibility and traceability reasons, it will take a longer and longer to migrate the old data as the application accumulates more of them. and it is also risky to edit/remove data if some part of the code still consumes them that you may not be aware of , so be extremely careful if you ever need to make changes to the core data structures.

Think before adding another field.

Sometime it feels much easy to simply add another field on the data model when a new requirement comes in, for instance, if we want to know if a certain model is modified since last time it is saved, it is easy to add a boolean flag on the model to reflect that, but the potential problem is that after 3 months, people might no longer remember what was this field used for and it becomes a tech debt when refactoring. Instead, if you could derive the state from the existing updated_at field before save to database to know if this data is modified.

Keeps your core business models from others models

Making a distinguish between the core data models from so-called data transfer models (DTOs) that is used to transfer data between layers and modules is a good practice, since the DTOs will likely change for different purposes with the refactoring of function argument and return types. However, the core data models should remain the same as long as the business domains that we are dealing with remain the same.

Maintain a consistent yet flexible interface

Another very challenge task most of the APIs has to overcome is to support multiple versions of the clients at the same time, an iOS application might take up to days or even weeks to pass the apple review and release a new version, the adoption rate will also slowly goes up unless you kill the older version completely. So in the most cases, we have to support older api schemas and delivery new functions with new schemas at the same time. the approach we took is to alway version our APIs like /api/v1/user /api/v2/user and define request and response objects separately (some ppl define serializers which is also fine) so we could easily adjust the client schema without affecting how the application works internally.

Testing

At current stage, we have full coverage on unit tests and integration (API) tests with a few UI tests as we still rely on QA to do regressions on every release. If you have limited engineering resource like us that build and maintain full coverage E2E tests is not feasible, I would recommend to focus on the quality of the unit tests as writing unit test is not only about the correctness of the program but more importantly it serves as “document” that each test case explains a situation that a function should cover, this is important because as the application get larger and more complicated, there will be more edge cases that will be easily forgotten and neglected, which is often the obstacle when refactoring the code because ppl fear any change could break existing behavior.

Constantly refactoring

No matter how careful we build the application, it will always get messier and harder to manage, not only because the accumulation of business logics but also the team is learning and discovering new way to solve certain problems, it is always necessary to spend time to revisit the existing code.

One thing we find to be quite fun is that we always allocate a few hours of the Friday afternoon to allow engineers to demo their PRs or proposals in a live session to all the engineers and explain the rational behind it, that way it encourages ppl to communicate and everyone would have chance to either learn from others or give some input.

Conclusion.

Building a solid software requires a engineering culture rather than a few practices, we should always pay attention to small things and keep rethinking our approach and make changes even if it does not have directly or immediately benefit to the product.