DEV Community: Myungwoo Song

Optimizing Ruby on Rails Serializers: Efficient Hash Unpacking in Serializer

Myungwoo Song — Sat, 02 Dec 2023 09:38:09 +0000

Introduction

In the world of Ruby on Rails development, serializers play a crucial role in transforming ActiveRecord objects into JSON formats suitable for client-side consumption. However, while invaluable, serializers can sometimes introduce performance bottlenecks. In this article, I share my experience in overcoming a specific challenge: efficiently unpacking results from a resource-intensive method.

The performance issues

Imagine a scenario common in high-traffic web applications, like an e-commerce platform, where efficiency in data handling is not just a luxury but a necessity. You have a method in your Rails model that needs to be invoked, and its results - a collection of various fields - must be flattened.

Here's the method in question:

# In a Rails Model
  # @return [Hash]
  #  [Number] total_paid_price
  #  [Number] total_unpaid_price
def total_book_prices
  query_result = book_suppliers.joins(book: :invoice)
                               .group("CASE WHEN invoices.stage < #{Invoice.stages[:approved]} THEN 'unpaid' ELSE 'paid' END")
                               .sum("invoices.total_price")
  {
    total_paid_price: query_result["paid"] || 0,
    total_unpaid_price: query_result["unpaid"] || 0,
  }
end

A naive approach might be to simply call this method twice within your serializer:

class SupplierListSerializer < ActiveModel::Serializer
  attributes :id, :title, :price, :author, ... , :total_paid_price, :total_unpaid_price

  def total_paid_price
    total_book_prices[:total_paid_price]
  end

  def total_unpaid_price
    total_book_prices[:total_unpaid_price]
  end
end

However, this leads to redundant database queries, a cardinal sin in performance-sensitive applications. Particularly with complex queries or large datasets, this inefficiency can significantly impact performance.

Solution I found

To circumvent this problem, I implemented a caching strategy within the serializer itself. By storing the result of the first method call in an instance variable, subsequent calls can use this cached result instead of hitting the database again. Here's how it looks:

class SupplierListSerializer < ActiveModel::Serializer
  attributes :id, :name, :number_of_books, ... , :total_paid_price, :total_unpaid_price

  ...

  def total_paid_price
    total_book_prices[:total_paid_price]
  end

  def total_unpaid_price
    total_book_prices[:total_unpaid_price]
  end

  private

  # Caching the result in an instance variable
  def total_book_prices
    @total_book_prices ||= object.total_book_prices
  end
end

This simple yet effective change ensures that our method total_book_prices` is called only once per serializer instance, significantly reducing database load and improving overall performance.

Conclusion

By implementing this caching technique, we've successfully navigated around one of the many performance pitfalls in Rails serializers. However, it's just the tip of the iceberg. There are other challenges, such as the infamous N+1 query issue, which can severely impact performance if not handled properly. In my next article, I'll delve into strategies to overcome this widespread issue.

Thank you for reading, and I encourage you to share your experiences or alternative solutions in the comments below!

Automating `updated_at` Timestamps with `update_all` in Ruby on Rails

Myungwoo Song — Thu, 16 Nov 2023 10:22:11 +0000

Introduction

Ruby on Rails greatly simplifies many programming tasks, yet sometimes its behavior can diverge from expectations. A notable example is when using the update_all method. This method, while efficient, does not automatically update the updated_at column in a database record. This oversight can lead to difficulties in tracking changes, particularly when auditing data history. This article presents a practical solution to this issue.

Understanding Ruby's Handling of Modules and Classes

Ruby's approach to modules and classes is fundamental to understanding the solution. In Ruby, a class can inherit features from a parent class and incorporate modules. This is achieved through a super pointer in each class, which references its parent, and a method_table that contains its methods. When a method is invoked, Ruby searches the method_table and, if necessary, proceeds up the inheritance chain. Including a module into a class inserts the module's methods into this lookup process, allowing for flexible and powerful customization.

The Art of Monkey Patching

'Monkey patching' refers to dynamically modifying or extending a class. In our case, we want Ruby to execute custom code before the native update_all method. This reverse lookup—checking our module first and then the original class—is made possible through the prepend keyword. By prepending our module, CustomUpdateAll, to ActiveRecord::Relation, we ensure that our version of update_all is found first. This technique is particularly useful in Rails, where code in config/initializer is executed after the framework has loaded.

Enhanced `update_all` Code

I first want to acknowledge that this is based on and inspired by Choncou's Code with some slight modifications of my own.

# config/initializer/custom_update_all.rb
module CustomUpdateAll
  def update_all(updates)
    # Check if updates should modify the updated_at column
    if updates.is_a?(Hash) && !updates.delete(:skip_touch) && column_names.include?("updated_at")
      super(updates.merge(updated_at: Time.zone.now)) # Merge our custom timestamp
    else
      super # Call the original update_all method
    end
  end
end

ActiveRecord::Relation.prepend(CustomUpdateAll)

Further Exploration

While this solution addresses hash-based updates, it can be expanded to handle strings and arrays. If you want, experiment and adapt the code to these formats, sharing your modifications and insights.

Conclusion

In summary, this article detailed a method to ensure updated_at is automatically updated when using update_all in Ruby on Rails. This solution enhances data integrity and simplifies record auditing. Your feedback and experiences with this approach are warmly welcomed.

References

Choncou's Code

Managing Ruby's Memory Bloating In Sidekiq: Strategies and Solutions

Myungwoo Song — Sat, 07 Oct 2023 07:27:30 +0000

Introduction

Ruby, a versatile programming language, is known for its ease of use and flexibility. However, it also comes with its own set of challenges, particularly related to memory management. In this article, we'll talk about Ruby's memory handling, the issues it can pose, and strategies to mitigate them.

Ruby's Memory Architecture

Ruby manages objects in two main places: Object Space and Memory Arena. Objects smaller than 24 bytes reside in Object Space, while larger objects are allocated to Memory Arena, with only their addresses stored in Object Space.

Ruby, implemented in C, allocates memory to Memory Arena using malloc in glib.c. To prevent concurrency issues, each Memory Arena has a lock. However, this can lead to performance bottlenecks in multi-threaded programs. To address this, Ruby creates multiple Memory Arenas and assign one Memory Arena to certain thread, but this can result in memory bloat.

Why we get the bloating?

But why? Think about Sidekiq with multiple threads. Maximum 1 Memory Arena will be assigned to each thread instead of each thread waiting for it's turn to access a Memory Arena. So, more than needed Memory have to be created, leading to more Memory usage.

Plus, there is another problem inside the allocator. It does not want to free the empty memory to the kernel. With these conditions altogether, We finally get the long lasting Memory Bloating!

Strategies to Address Ruby's Memory Issues

So, how can we deal with them? There are several strategies to tackle these memory problems:

Changing Memory Allocator: Consider switching from the default malloc allocator to an alternative like jemalloc. Note that compatibility issues may arise on Alpine-based systems.
Setting Memory Arena Limits: You can set a tighter limit on the number of Memory Arenas using MALLOC_ARENA_MAX = 2. This can help mitigate concurrency-related performance issues.
Freeing Memory Not In Use: Another potential solution is to address Ruby's reluctance to free memory after use. Check out this insightful article for more information on this approach.
Rerunning the Program: There might be some companies hesitating to modify the magical environment variable using Apline-based docker image just like my case. Then, periodically restarting the program can be a solution too. I will concentrate on this solution below.

Rerunning the Program as a Temporary Fix

In some cases, a quick workaround involves periodically restarting your Ruby application. This can temporarily alleviate memory issues for especially those who are hesitating to manipulate the magical environment variable. However, it's not a fundamental solution.

Memory Challenges and Job Retries

When dealing with Sidekiq containers, shutting them down can lead to unexpected job retries. Whenever a Sidekiq process shuts down, the jobs it was working on return to the Redis queue. When the process restarts, it reprocesses all the jobs, regardless of their previous progress.

To address this side effect, consider implementing a generous shutdown function in your logic. This function ensures that jobs in progress are not retried unnecessarily. It's a crucial addition to managing memory issues in Sidekiq.

Implementing the Memory Management Solution

Here's a high-level overview of the approach:

Utilize AWS EventBridge to run a function every 5-10 minutes.
The function uses the AWS SDK to monitor the running-time of processes.
The function pauses processes that have been running for an extended period using the TSTP signal.
After the process completes its last job, the function terminates it using the TERM signal.

Additional considerations include maintaining at least one running process, limiting the maximum number of processes, and implementing scaling logic.

Conclusion

Ruby's memory management challenges are a common issue for developers, but they are not insurmountable. By understanding the fundamentals of Ruby's memory handling and implementing appropriate strategies, you can optimize the performance of your Ruby applications. If you've encountered similar challenges or have questions, feel free to share your experiences and seek assistance within the community.

References

Thank you for reading!

DEV Community: Myungwoo Song

Optimizing Ruby on Rails Serializers: Efficient Hash Unpacking in Serializer

Introduction

The performance issues

Solution I found

Conclusion

Automating `updated_at` Timestamps with `update_all` in Ruby on Rails

Introduction

Understanding Ruby's Handling of Modules and Classes

The Art of Monkey Patching

Enhanced update_all Code

Further Exploration

Conclusion

References

Managing Ruby's Memory Bloating In Sidekiq: Strategies and Solutions

Introduction

Ruby's Memory Architecture

Why we get the bloating?

Strategies to Address Ruby's Memory Issues

Rerunning the Program as a Temporary Fix

Memory Challenges and Job Retries

Implementing the Memory Management Solution

Conclusion

References

Enhanced `update_all` Code