DEV Community: Lazy coder

Streamlining the Data Symphony: Enhancing Serialization in Ruby on Rails

Lazy coder — Fri, 02 Aug 2024 10:53:00 +0000

Streamlining the Data Symphony: Enhancing Serialization in Ruby on Rails

In the intricate ballet of web application development, Ruby on Rails often leads the performance with grace. But when it comes to the grand orchestration of large, nested data structures, even the most agile framework can stumble. Let's embark on a journey to refine the process of serializing complex data, like the multi-layered hierarchy of classrooms, students, lessons, exercises, and scores.

Prologue: The Nested Data Conundrum

Picture a vast library where each bookshelf represents a layer of data. The deeper you venture, the more intricate the connections become. Fetching such data in one go can be as daunting as navigating a labyrinth. But fear not, for we have strategies to illuminate the path.

Act I: The Cache – A Treasure Trove of Pre-Rendered Delights

When the same data is requested repeatedly, caching becomes our trusty time machine, transporting us back to a moment when the data was already prepared. Implementing cache at various levels—page, action, or fragment—allows us to serve data swiftly, reducing the load on our server and speeding up response times.

Technical Insight:

Use Rails' built-in caching mechanisms or integrate with Redis for more granular control.
Cache scores and exercises, as they are the most dynamic elements in our data structure.

Act II: The Serializer – Crafting with Precision

Just as a master craftsman selects the finest tools, we choose Oj for its speed and efficiency in shaping our JSON output. Oj is a gem that stands out for its performance, making it an ideal choice for our serialization needs.

Technical Insight:

Add gem 'oj' to your Gemfile.
Replace JSON.generate with Oj.dump for faster serialization.

Act III: Eager Loading – The Art of Anticipation

In the bustling kitchen of our application, eager loading is the sous-chef who prepares all the ingredients in advance. This technique prevents the common N+1 query problem, ensuring that our data is fetched in the most efficient manner possible.

Technical Insight:

Use .includes with ActiveRecord to preload associated data.
Example: ClassRoom.includes(students: {lessons: {exercises: :scores}}).where(teacher_id: 1)

Act IV: Asynchronous Processing – The Ensemble of Background Tasks

As in a symphony where each instrument plays independently yet harmoniously, asynchronous processing allows different parts of our data to be prepared in parallel. Sidekiq is our conductor, orchestrating these background tasks to perfection.

Technical Insight:

Set up Sidekiq to handle data serialization tasks outside the main request/response cycle.
Create Sidekiq workers to serialize data and store it for subsequent requests.

Act V: Async Programming – The Power of Parallelism

Rails' load_async, in conjunction with Concurrent::Async, is like having multiple artists painting different sections of our mural simultaneously. This parallelism ensures that our CPU is utilized to its fullest potential, speeding up the overall process.

Technical Insight:

Utilize load_async for asynchronously loading ActiveRecord relations.
Example: ClassRoom.find(1).load_async.students.load_async.lessons

Epilogue: The Road Ahead

As we conclude our journey, we recognize that the path to optimization is a continuous exploration. The strategies we've discussed are but a few arrows in our quiver. As we venture further, we might discover new frameworks like Hanami, which offer built-in solutions for concurrency challenges.

Further Adventures:

Experiment with pagination to limit data depth and reduce payload size.
Employ data compression techniques to enhance API response efficiency.
Use monitoring and profiling tools to identify bottlenecks and refine performance.

By striking a balance between storytelling and technical guidance, we've crafted an article that not only educates but also engages. The art of serialization is a dance of efficiency and performance, and with the right steps, we can ensure that our data structures flow as smoothly as a well-written symphony.

irbrc to help with your efficiency

Lazy coder — Wed, 24 Jul 2024 19:16:35 +0000


IRB.conf[:PROMPT][:DEV] = { # name of prompt mode
  :AUTO_INDENT => false,          # disables auto-indent mode
  :PROMPT_I =>  ">> ",            # simple prompt
  :PROMPT_S => nil,               # prompt for continuated strings
  :PROMPT_C => nil,               # prompt for continuated statement
  :RETURN => "    ==>%s\n"        # format to return value
}

#IRB.conf[:PROMPT_MODE] = :MY_PROMPT

# Tracking and Debugging Helpers

# Log method calls with their arguments
# Usage: track_method_calls(User, :save)
def track_method_calls(klass, method_name)
  klass.class_eval do
    old_method = instance_method(method_name)
    define_method(method_name) do |*args, &block|
      puts "Calling #{klass}##{method_name} with args: #{args.inspect}"
      old_method.bind(self).call(*args, &block)
    end
  end
end

# Track SQL queries
# Usage: track_sql_queries
def track_sql_queries
  ActiveSupport::Notifications.subscribe('sql.active_record') do |*args|
    event = ActiveSupport::Notifications::Event.new(*args)
    puts "SQL Query: #{event.payload[:sql]}"
    puts "Duration: #{event.duration.round(2)} ms"
    puts "---"
  end
end

# Count method calls
# Usage: count_method_calls(User, :update)
def count_method_calls(klass, method_name)
  count = 0
  klass.class_eval do
    old_method = instance_method(method_name)
    define_method(method_name) do |*args, &block|
      count += 1
      puts "#{klass}##{method_name} called #{count} times"
      old_method.bind(self).call(*args, &block)
    end
  end
end

# Track object allocations
# Usage: track_allocations { your_code_here }
def track_allocations
  require 'objspace'
  GC.disable
  before = ObjectSpace.count_objects
  yield
  after = ObjectSpace.count_objects
  puts "Object allocations:"
  after.each do |key, value|
    diff = value - before[key]
    puts "  #{key}: #{diff}" if diff != 0
  end
ensure
  GC.enable
end

# Profile code execution
# Usage: profile_code { your_code_here }
def profile_code
  require 'ruby-prof'
  RubyProf.start
  yield
  result = RubyProf.stop
  printer = RubyProf::FlatPrinter.new(result)
  printer.print(STDOUT)
end

# Track method execution time
# Usage: track_time(User, :expensive_method)
def track_time(klass, method_name)
  klass.class_eval do
    old_method = instance_method(method_name)
    define_method(method_name) do |*args, &block|
      start_time = Time.now
      result = old_method.bind(self).call(*args, &block)
      end_time = Time.now
      puts "#{klass}##{method_name} took #{(end_time - start_time) * 1000.0} ms"
      result
    end
  end
end

# Log instance variable changes
# Usage: track_ivar_changes(user, :name)
def track_ivar_changes(object, ivar_name)
  object.define_singleton_method("#{ivar_name}=") do |value|
    old_value = instance_variable_get("@#{ivar_name}")
    instance_variable_set("@#{ivar_name}", value)
    puts "#{object.class}##{ivar_name} changed from #{old_value.inspect} to #{value.inspect}"
  end
end


# Memory usage tracker
# Usage: track_memory { your_code_here }
def track_memory
  memory_before = `ps -o rss= -p #{Process.pid}`.to_i
  yield
  memory_after = `ps -o rss= -p #{Process.pid}`.to_i
  puts "Memory usage increased by #{memory_after - memory_before} KB"
end


def compare_methods(count = 1, *methods)
  require 'benchmark'
  Benchmark.bmbm do |b|
    methods.each do |method|
      b.report(method) { count.times { send(method) } }
    end
  end
  nil
end

def pp_hash(hash, indent=1)
  hash.each do |k, v|
    print "  " * indent
    if v.is_a?(Hash)
      puts "#{k}:"
      pp_hash(v, indent+1)
    else
      puts "#{k}: #{v}"
    end
  end
end

def measure_allocation
  before = GC.stat(:total_allocated_objects)
  yield
  after = GC.stat(:total_allocated_objects)
  puts "Allocated objects: #{after - before}"
end

def http_get(url)
  require 'httparty'
  response = HTTParty.get(url)
  puts "Status: #{response.code}"
  puts "Body: #{response.body}"
end

def explore_relation(relation)
  puts "SQL: #{relation.to_sql}"
  puts "Explained:"
  puts relation.explain
end

def parse_json(json_string)
  require 'json'
  JSON.pretty_generate(JSON.parse(json_string))
end

def list_constants(mod)
  mod.constants.sort.each do |const|
    puts "#{const} = #{mod.const_get(const)}"
  end
end

# Usage: class_hierarchy(Array)
def class_hierarchy(klass)
  hierarchy = [klass]
  while (klass = klass.superclass)
    hierarchy << klass
  end
  hierarchy.each_with_index do |k, i|
    puts "#{' ' * i}#{k}"
  end
end

# Usage: method_info("hello", :upcase)
def method_info(obj, method_name)
  method = obj.method(method_name)
  puts "Name: #{method.name}"
  puts "Owner: #{method.owner}"
  puts "Parameters: #{method.parameters}"
  puts "Arity: #{method.arity}"
  puts "Source Location: #{method.source_location&.join(':')}"
end

# Usage: set_breakpoint(User, :save)
def set_breakpoint(klass, method_name)
  require 'pry'
  klass.send(:define_method, method_name) do |*args, &block|
    binding.pry
    super(*args, &block)
  end
end

# Usage: list_instance_variables(some_object)
def list_instance_variables(obj)
  obj.instance_variables.each do |var|
    puts "#{var}: #{obj.instance_variable_get(var).inspect}"
  end
end

# Usage: show_ancestors(Array)
def show_ancestors(klass)
  puts "Ancestors:"
  klass.ancestors.each { |ancestor| puts "  #{ancestor}" }
  puts "\nIncluded Modules:"
  (klass.ancestors - klass.superclass.ancestors - [klass]).each { |mod| puts "  #{mod}" }
end

# Usage: list_constants
def list_constants
  Module.constants.sort.each do |constant|
    puts constant
  end
end

# Usage: explore_methods("hello")
def explore_methods(obj)
  methods = {
    instance: obj.methods - Object.methods,
    inherited: obj.methods & Object.methods,
    singleton: obj.singleton_methods
  }

  methods.each do |category, method_list|
    puts "#{category.to_s.capitalize} methods:"
    method_list.sort.each { |method| puts "  #{method}" }
    puts
  end
end

# Usage: show_method_source(Array, :map)
def show_method_source(obj, method_name)
  require 'pry'
  obj.method(method_name).source.display
end

# Usage: list_classes
def list_classes
  Object.constants
    .select { |c| Object.const_get(c).is_a? Class }
    .sort
    .each { |klass| puts klass }
end

# Usage: method_location(Array, :map)
def method_location(obj, method_name)
  file, line = obj.method(method_name).source_location
  puts "Defined in #{file}:#{line}"
end

# Usage: subclasses_of(ActiveRecord::Base)
def subclasses_of(klass)
  ObjectSpace.each_object(Class).select { |k| k < klass }
end

def track_changes(target, method_name)
  original_value = target.send(method_name.to_s.sub('=', ''))
  puts "Starting to track changes to #{target}.#{method_name}"
  puts "Initial value: #{original_value.class}"

  trace = TracePoint.new(:line) do |tp|
    #next unless tp.method_id.to_s == method_name.to_s && tp.self.instance_of?(target.class) && tp.self == target
    next unless tp.self.instance_of?(target.class) && tp.self == target && tp.method_id.to_s.include?(method_name.to_s.sub('=', ''))

    current_value = target.send(method_name.to_s.sub('=', ''))
    if current_value != original_value
      puts "\nChange detected in #{target}.#{method_name}:"
      puts "  From: #{original_value.class}"
      puts "  To:   #{current_value.class}"
      puts "  At:   #{tp.path}:#{tp.lineno}"
      puts "  Backtrace:"
      puts caller[0..5].map { |line| "    #{line}" }
      puts '-'*60
      original_value = current_value
    end
  end

  trace.enable
  puts "Tracking enabled. Run 'trace.disable' to stop tracking."

  trace  # Return the trace object so it can be disabled later
end

def track_method(klass, method_name, process=nil)
  trace = TracePoint.new(:call) do |tp|
    next unless tp.method_id.to_s == method_name.to_s

    next unless (klass.is_a?(Class) ? tp.self.is_a?(klass) : tp.self == klass)

    puts '-'*20
    puts "#{tp.path}:#{tp.lineno}"
    puts "#{tp.defined_class}##{tp.method_id}" if tp.defined_class
    process.call if process
  end

  trace.enable
  if block_given?
    yield
    trace.disable
  else
    puts "Tracking enabled. Run 'trace.disable' to stop tracking."
  end

  trace  # Return the trace object so it can be disabled later
end

N+1 in Ruby on Rails

Lazy coder — Thu, 15 Jun 2023 22:52:14 +0000

How to detect N+1

Prosopite

Install

gem install prosopite

bundle add prosopite

# Gemfile
gem 'prosopite'

Configure

Controller
Add the following to ApplicationController

class ApplicationController < ActionController::Base
  unless Rails.env.production?
    before_action do
      Prosopite.scan
    end

    after_action do
      Prosopite.finish
    end
  end
end

Enable logging

# config/environments/development.rb

config.after_initialize do
  Prosopite.rails_logger = true
end

Spec

# config/environments/test.rb

config.after_initialize do
  Prosopite.rails_logger = true
  Prosopite.raise = true
end

# spec/spec_helper.rb

config.before do
  Prosopite.scan
end

config.after do
  Prosopite.finish
end

Now everytime you run the test, you will be able to see any error about N+1 queries.

Note: It won't raise any exception if only one association.

For example

class Teacher < ActiveRecord
  has_many :students
end

class Student < ActiveRecord
  belongs_to :teacher
end

No exception

teacher = Teacher.create(name: 'John')
teacher.students.create(name: 'Tom')

Teacher.first.students do |student|
  puts student.name
end

Throw exception about N+1

teacher = Teacher.create(name: 'John')
teacher.students.create(name: 'Tom')
teacher.students.create(name: 'Jerry')

Teacher.first.students do |student|
  puts student.name
end

Prosopite::NPlusOneQueriesError:
       N+1 queries detected:
         SELECT `students`.* FROM `students` WHERE `students`.`id` = ? LIMIT ?

Convert object to Yaml by loading js lib

Lazy coder — Thu, 15 Jun 2023 22:08:57 +0000

Convert object to Yaml


$.getScript('https://unpkg.com/js-yaml@4.1.0/dist/js-yaml.min.js');
copy(jsyaml.dump(myObject));

Convert object to JSON

copy(JSON.stringify(myObject));

How to debug image build with Dockerfile

Lazy coder — Thu, 15 Jun 2023 22:04:57 +0000

How to debug image build with Dockerfile

Dockerfile to build the env for Ruby on Rails App

FROM ruby
# install essential libs/commands
RUN apt-get update && apt-get install lib-ssl
...
# install gems
RUN bundle install
...
# install npm
RUN yarn install
COMMAND bundle exec rails server

delete the following part causing errors

FROM busybox
RUN echo 'hello world' > /tmp/test

RUN exit 1
....
RUN echo 'ready'

~~RUN exit 1~~
Just remove RUN exit 1 and the following

run intermediate image with sha Turn off buildkit to get the sha for each layer docker run --rm -it current sh to debug

❯ DOCKER_BUILDKIT=0 docker build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM busybox
 ---> abaa813f94fd
Step 2/3 : RUN echo 'hello world'
 ---> Using cache
 ---> 551ba8324834
Step 3/3 : RUN exit 1
 ---> Running in 5e3c523c38a3
The command '/bin/sh -c exit 1' returned a non-zero code: 1

# pick sha of last successful layer
docker run --rm -it 551ba8324834 sh

nsenter to debug

Enter into the name space of the process.
Namespaces are a feature of the Linux kernel that partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources.

add RUN sleep infinite to Dockerfile
docker run -it --rm --privileged --pid=host justincormack/nsenter1 to get to the building host ¹
ps -ef|grep sleep to find pid
nsenter -p -m -u -i -n -t 10012 sh

FROM busybox
RUN echo 'hello world'
RUN sleep infinite
RUN exit 1

❯ docker build -t test  .
[+] Building 10.7s (5/7)
 => [internal] load build definition from Dockerfile                                                                                             0.0s
 => => transferring dockerfile: 73B                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                0.0s
 => [1/4] FROM docker.io/library/busybox                                                                                                         0.0s
 => CACHED [2/4] RUN echo 'hello world'                                                                                                          0.0s
 => [3/4] RUN sleep infinite

use target to build sucessfully the image multiple-stage builds

FROM busybox as working
RUN echo 'hello world'

FROM working as error
RUN exit 1

# build image with target
docker build -t test --target working .

# run image to debug
docker run --rm -it test sh

docker run --privileged --pid=host -it alpine \
nsenter -t 1 -m -u -n -i sh

--rm : removes the container after it is stopped -ti (or -t -i) : adds a tty and leaves the standard input opened
--privileged : grants additional permissions to the container, it allows the container to gain access to the devices of the host (/dev)
--pid=host : allows the containers to use the processus tree of the Docker host (the VM in which the Docker daemon is running)

check pid

❯ docker run -ti --rm busybox sh
❯ docker ps
❯ docker inspect --format '{{.State.Pid}}' a57c56a83e54
18762

althernative way using alpine and nsenter command ↩