DEV Community: Sang Huynh Thanh

Sidekiq vs GoodJob: A Comprehensive Comparison

Sang Huynh Thanh — Mon, 07 Apr 2025 08:35:45 +0000

In the Rails ecosystem, job processing libraries are essential for managing background tasks efficiently. We will compare their features, usability, and suitability for different project needs.

Documentation Overview

GoodJob offers a single README.md file that is densely packed with information, which may overwhelm some users. In contrast, Sidekiq utilizes GitHub's Wiki feature, providing easier navigation and topic-specific searches. While GoodJob's documentation is comprehensive, it can be challenging to find specific information compared to the better readability and ease of use found in Sidekiq.

Batches and Callbacks

When it comes to batch processing, Sidekiq and GoodJob take distinctly different approaches:

Sidekiq excludes batch features from its free version. Users must upgrade to Pro or Enterprise plans to access this functionality, which may pose a barrier for smaller projects or teams with limited budgets.
GoodJob includes batch processing by default, allowing users to implement this feature without additional costs. This community-driven gem eliminates the need for extra plugins or upgrades.

Features Comparison

Definition: In Sidekiq Pro, batches refer to collections of jobs monitored as a group. You can create a set of jobs to execute in parallel and then execute a callback when all jobs are finished. GoodJob tracks a set of jobs and enqueues an optional callback job when all the jobs have finished (succeeded or discarded).
Jobs can be added to an existing batch? Yes for both.
Access batch information: Yes for both.
Callback with statuses: Sidekiq uses statuses like complete, success, and death. GoodJob uses finish, success, and discard.
Linger and Expiration: Sidekiq's batch data lingers in Redis for 24 hours and expires after 30 days. In contrast, GoodJob ensures that batch information persists in the database before processing.
Scheduled Jobs: Sidekiq does not have built-in support for scheduled jobs without upgrading. GoodJob supports both CLI and async execution modes and includes a dedicated Cron tab for managing scheduled tasks.

Web UI Management

Both libraries offer web interfaces with distinct features:

Common Features

Support authentication and authorization.
Offer a polling feature for job status updates.
Provide comprehensive dashboards for monitoring job queues.
Allow micromanagement capabilities such as retrying and discarding jobs.

Unique Features

Authentication: Sidekiq supports many identity providers (e.g., Google, GitHub), while GoodJob offers basic authentication with role-based access.
Filtering: Sidekiq does not support filtering from the UI, whereas GoodJob provides various filtering options by queue, job type, and parameters.
Process or Thread Information: Available in Sidekiq but not in GoodJob.
Reschedule Job?: Yes in GoodJob; no in Sidekiq.

Error Handling and Dead Job Management

Both libraries effectively handle errors during job execution:

Both catch exceptions during job execution and log errors with details.
They support automatic retries on failure with configurable limits.
Both integrate with Rails logger for customizable logging.

Dead Job Handling

Sidekiq moves failed jobs to a separate "Dead Job Queue" after exhausting all retry attempts and automatically prunes dead jobs after six months.
GoodJob keeps dead jobs in the same database as other job records, making them easier to manage using standard SQL tools. It allows custom error handling through ActiveJob callbacks.

Job Prioritization

Sidekiq

Allows multiple queues for effective priority management.
Jobs are processed based on queue priority; within a single queue, they follow FIFO (First In, First Out) order.

GoodJob

Supports multiple queues with a priority attribute for granular control.
Considers both queue and job priority when selecting which job to run next.
In both libraries, a higher-priority queue will process its jobs before lower-priority queues. However, this strategy can lead to potential issues; if a job within a high-priority queue is suspended or fails, it can cause all lower-priority queues to wait indefinitely.
Within a single queue, jobs with higher priority will execute before those with lower priority. This feature provides more granular control over job execution timing, allowing less urgent tasks to wait until resources are available.

Data Retention and Cleanup

GoodJob uses a database for long-term storage with configurable retention periods. In contrast, Sidekiq primarily uses Redis, which automatically expires data after completion.

Key Differences

Storage Mechanism: Sidekiq relies on ephemeral Redis storage while GoodJob uses a database.
Historical Data Management: GoodJob is better suited for maintaining longer-term job history compared to Sidekiq's focus on current tasks.
Configuration Flexibility: GoodJob offers more detailed control over job record retention compared to Sidekiq's simpler setup.

Conclusion

Both Sidekiq and GoodJob offer robust job processing capabilities but cater to different needs within the Ruby on Rails ecosystem:

Sidekiq is ideal for teams seeking straightforward setups willing to invest in premium features. Its reliance on Redis allows quick access but limits long-term data retention.
GoodJob, embracing an open-source philosophy, provides comprehensive features without additional costs. Its use of PostgreSQL allows better long-term data management, making it suitable for applications requiring detailed historical insights.

Ultimately, your choice between Sidekiq and GoodJob should align with your project requirements, budget considerations, and desired level of control over job processing and data management. Evaluate your unique needs to determine which library fits best with your workflow and objectives.

Grafana K6: Load testing for awareness of your infrastructure

Sang Huynh Thanh — Mon, 07 Apr 2025 08:19:38 +0000

The versatile open-source tool for robust load testing. Learn how to stress-test your systems and build scalable applications that can handle real-world traffic demands.

What is Grafana K6?

Grafana K6 is a modern load-testing tool that allows developers and QA engineers to test the reliability and performance of their systems. It's designed to catch performance regressions and problems early in the development cycle, helping teams build applications that can scale effectively.

Key Features:

Minimal Resource Consumption: Optimized for running high-load tests efficiently.
JavaScript-based Scripting: Uses ES6 modules, allowing for modular and reusable test scripts.
Flexible Configuration: Supports various load patterns and test scenarios.
Extensible: Can be integrated with other tools and platforms for comprehensive testing and monitoring.

Getting Started with K6

Installation

For macOS users, K6 can be easily installed. (Note: Installation instructions for other operating systems can be found on the official K6 documentation.)

Understanding K6 Terminology

VUs (Virtual Users): Simulated users generating traffic in your tests. More VUs generally mean more simulated traffic.

K6 runs multiple iterations in parallel using virtual users. Each VU is essentially a while(true) loop, executing your test script repeatedly for the duration of the test.

K6 Test Lifecycle

A typical K6 test goes through several stages:

Initialization: Setting up test configuration and resources.
Setup: Preparing the test environment (optional).
Execution: Running the actual test scenarios.
Teardown: Cleaning up after the test (optional).

Types of Load Tests with K6

K6 supports various types of load tests, each designed to assess different aspects of your system's performance:

1. Average-Load Testing

Purpose: To understand if your system meets performance goals under normal conditions.

When to use:

Determine the average workload your system can handle
Verify performance standards after system changes

Example Configuration:

export const options = {
  stages: [
    { duration: '5m', target: 100 }, // Ramp-up
    { duration: '30m', target: 100 }, // Sustain
    { duration: '5m', target: 0 }, // Ramp-down
  ],
};

Example result:

2. Stress Testing

Purpose: Assess system performance under heavier-than-usual loads.

When to use:

After running average-load tests
To verify stability during peak usage periods (e.g., holidays, special events)

Example Configuration:

export const options = {
  stages: [
    { duration: '10m', target: 200 }, // Ramp-up
    { duration: '30m', target: 200 }, // Sustain
    { duration: '5m', target: 0 }, // Ramp-down
  ],
};

Tip: Reuse your average-load test script, but increase the load or VU count.

3. Soak Testing

Purpose: Check for performance degradation over extended periods.

Considerations:

Longer duration (e.g., 3, 8, 24, or even 72 hours)
Monitor backend resources (RAM, CPU, Network, cloud resource growth)

Example Configuration:

export const options = {
  stages: [
    { duration: '5m', target: 100 }, // Ramp-up
    { duration: '8h', target: 100 }, // Sustain
    { duration: '5m', target: 0 }, // Ramp-down
  ],
};

4. Spike Testing

Purpose: Simulate sudden, massive increases in traffic.

Characteristics:

Rapid increase to extremely high loads
Brief or no plateau period
Fast ramp-down

Example Configuration:

export const options = {
  stages: [
    { duration: '2m', target: 2000 }, // fast ramp-up to a high point
    // No plateau
    { duration: '1m', target: 0 }, // quick ramp-down to 0 users
  ],
};

Considerations:

Tests may not be completed due to system overload
Useful for assessing recovery times and behavior during critical overloads

Best Practices and Tips

Use Tags: Group metrics for dynamic URLs to improve analysis.
Combine Criteria: Utilize Checks and Thresholds for comprehensive testing.
Choose the Right Load Model: Decide between modeling load by VU count or iterations per second.
Keep it Simple: Stick to straightforward load patterns (ramp-up, plateau, ramp-down) for clear results.
Diversify Your Tests: No single test type eliminates all risks, so use a combination of testing methods.

Conclusion

Grafana K6 is a versatile and powerful tool for load-testing modern applications. By incorporating various testing methods – from average-load to spike testing – you can ensure your systems are robust, scalable, and ready to handle real-world traffic patterns. Remember, the key to effective load testing is not just running the tests but also analyzing the results and continuously improving your system based on those insights.

Start integrating K6 into your development and QA processes today, and take a significant step towards building more resilient and performant applications.

Kamal-Proxy: Simplifying Zero-Downtime Deployments

Sang Huynh Thanh — Mon, 07 Apr 2025 07:24:39 +0000

Discover how the custom reverse proxy enhances Kamal’s deployment strategy with seamless traffic routing, zero-downtime updates, and automated security measures for containerized application.

Introduction to Kamal-Proxy

Kamal, a straightforward deployment tool built around Docker containers, has introduced its custom proxy called Kamal-Proxy. This new component aims to simplify the deployment process, particularly for newcomers or those less familiar with complex technological setups. In this post, we'll explore what Kamal-Proxy is and why it's an essential part of Kamal's deployment strategy.

What is Kamal-Proxy?

Kamal-Proxy is a custom-built reverse proxy designed specifically for Kamal's deployment needs. It serves as a critical component in managing traffic to deployed applications, handling multiple services, and ensuring smooth, zero-downtime deployments. Unlike its predecessor Traefik, Kamal-Proxy is tailored to work seamlessly with Kamal's imperative approach to deployments.

Why does Kamal need a proxy like Kamal-Proxy?

Kamal-Proxy addresses these gaps by offering key features:

Simplicity in Service Deployment: Kamal-Proxy maintains Kamal’s ease-of-use, allowing users to deploy services by specifying the service type and target without the need for complex configurations.
Zero-Downtime Deployments: While Kamal's basic approach doesn’t include automatic container replacement, Kamal-Proxy ensures zero-downtime updates by intelligently routing traffic during deployments, minimizing service disruption.
Traffic Management and Multi-Service Support: Kamal-Proxy intelligently routes incoming requests to the appropriate containers, even when hosting multiple applications on the same server. It eliminates port conflicts by using hostnames for routing.
Health Monitoring: To ensure reliable service, Kamal-Proxy continuously monitors container health, directing traffic only to healthy instances.
Automatic TLS Certificates: Security is enhanced with automatic issuance and management of TLS certificates, ensuring encrypted communication without manual intervention by utilizing Let’s Encrypt.

Kamal-Proxy Functionalities

Kamal-Proxy integrates seamlessly with Kamal's deployment process:

Traffic Interception: It intercepts incoming client requests.
Intelligent Routing: Based on predefined criteria (like hostnames), it routes requests to the appropriate containers.
Update Management: During deployments, it gradually shifts traffic from old to new containers, ensuring continuous service.
Health-Based Routing: By monitoring container health, it automatically adjusts routing to maintain service reliability.

Imperative vs. Declarative?

Declarative (Traefik):

Specifies the desired end state (expected result), and the system figures out how to achieve that state.

# You declare what you want, not how to achieve it

http:
  routers:
    my-router:
      rule: "Host(`example.com`)"
      service: my-service
services:
  my-service:
    loadBalancer:
      servers:
        - url: "http://localhost:8080"

Imperative (Kamal-Proxy):

Provide step-by-step instructions to achieve the desired outcome.
You have more direct control over each action.

# You explicitly state each action to take

kamal proxy:add-route example.com --target localhost:8080
kamal proxy:enable-ssl example.com

Removal of Host-Level Complexities

Kamal-Proxy eliminates the need for "special cord files" that Traefik required:

The Problem: Traefik, being declarative, needed a way to understand the health status of containers.
The Workaround: Kamal had to create special files (referred to as "cord files") on the host system.
- These files acted as a bridge between Kamal's understanding of container health and Traefik's configuration.
- Example: A file might be created or removed to signal whether a container is healthy or not.
The Drawback: This solution was seen as inelegant and added complexity to the host system.

This change simplifies the system architecture and makes deployments more straightforward, benefiting both new and experienced users.

Conclusion

Kamal-Proxy represents a significant advancement in Kamal's deployment toolkit. By providing a custom-built proxy solution that aligns perfectly with Kamal's philosophy of simplicity and direct control, it enhances the deployment experience for users of all expertise levels. From simplifying service management to ensuring zero-downtime updates and automatic security measures, Kamal-Proxy embodies Kamal's commitment to making container deployment accessible and efficient.

Understanding Kamal-Proxy is key to leveraging the full potential of Kamal for your deployment needs, offering a streamlined, secure, and user-friendly approach to managing containerized applications in today's dynamic digital landscape.

How to generate a PDF file in Rails using the Grover gem?

Sang Huynh Thanh — Sat, 08 Apr 2023 04:03:42 +0000

How to generate a PDF file in Rails using the Grover gem?

When it comes to generating PDF files from HTML templates, there are several libraries and tools available. In this article, we'll take a closer look at the Grover gem and discuss why we chose it over other alternatives like Prawn and wicked_pdf.

A modern way to Generate PDF Files in Rails

Challenges with wicked_pdf

Initially, we used wicked_pdf in some of our ongoing projects, and it worked well in most cases, enabling us to generate PDF files from our templates successfully. However, we did face some challenges with certain Bootstrap features like the col and row and flexbox styles, which were not fully supported by the library. We were able to work around this issue by changing the current view implementation to using a table instead of a flexbox or grid. However, this is not an ideal solution, as it limits the flexibility of the view.

Additionally, we discovered that wicked_pdf is built on top of wkhtmltopdf, an open-source command-line tool that uses the Webkit rendering engine to convert HTML files to PDF and various image formats. However, this tool was archived as of January 2, 2023, which made us reconsider our choice of using wicked_pdf.

Introducing Grover

This led us to explore other PDF generation tools and libraries, and we discovered the Grover gem. Grover is a Ruby gem that uses Google Puppeteer and Chromium to convert HTML into PDF, PNG, or JPEG files. To use Grover, you will need to install both the Grover gem and the puppeteer library.

Advantages of Grover and also limitations of wicked_pdf

One of the key advantages of Grover is its reputation and reliable community. It has been widely adopted by the Rails community due to its ease of use and versatility. Grover supports the latest CSS3 features and JavaScript libraries, making it a popular choice for complex PDF conversions.

Another advantage of Grover is its ability to render dynamic content, which is crucial for web applications that generate documents on the fly. This is achieved through Google Puppeteer, which provides a headless Chrome/Chromium environment to execute JavaScript and render dynamic content.

In contrast, wicked_pdf is a popular alternative for PDF generation in Rails, but it has limitations in terms of supporting the latest CSS3 and JavaScript features. Additionally, it may require additional configuration and setup for rendering dynamic content, which could be time-consuming.

Customization options

Grover offers a range of configuration options to control the PDF layout, including page size, orientation, margin, and header/footer content. Moreover, it allows you to inject custom CSS and JavaScript code into the HTML to further customize the PDF output. This provides a high degree of flexibility in terms of customization.

Methods for Generating PDF Files using Grover in Rails

Generating PDF files in Rails is a common task, and Grover is one of the gems that can help us accomplish it. In this blog post, we will explore two different methods for generating PDF files using Grover in Rails.

Render from template

Rendering a template is a common approach to generating PDF files in Rails. This method involves creating an HTML template that includes placeholders for dynamic data or a custom template without a header or side navigation. Grover can then merge the template with the data to generate a PDF file.

One advantage of this approach is that it provides complete control over the layout and formatting of the PDF output. This approach is particularly useful for generating documents such as invoices, receipts, and reports that require a specific layout and design.

To render a template with Grover, you can use the render_to_string method in the Rails controller. This method takes a view template and any necessary data and returns the HTML code as a string. However, this code only contains relative URLs for stylesheets, images, and other assets. As a result, there may be issues with missing stylesheets and fonts when generating PDFs using Grover. In such cases, it is important to set the display_url property correctly. This is because Grover defaults to using example.com as the host, which can cause errors when attempting to access stylesheets and fonts from other sites.

To prevent or resolve this issue, pass the relative HTML code as a string, along with any necessary data, to the Grover::HTMLPreprocessor helper. This will convert it into HTML code with all absolute URLs.

relative_html = PayslipController.new.render_to_string({
  template: 'controller/view',
  layout: 'my_layout',
  locals: { payroll: payroll, address: address }
})

absolute_html = **Grover::HTMLPreprocessor.process**(relative_html, 'http://my.server/', 'http')

We also have the second option that involves specifying a display_url that will replace the default configuration of the example.com host, which can cause the CORS error when the page is displaying on another host and is different from our host.

pdf = Grover.new(relative_html, **display_url: base_url**).to_pdf

Rendering from a URL

This approach involves specifying a URL to a web page along with the user’s cookies to make sure that Grover has permission to access the page we need to print. This method is useful for generating PDFs from existing web pages, such as product pages, blog posts, or help documentation.

Grover.new(url, cookies: header_cookies).to_pdf

def header_cookies
  request.headers['Cookie'].split('; ').map do |cookie|
    key, value = cookie.split '='

    { name: key, value: value, domain: request.headers['Host'] }
  end
end

Comparison, which one should I use?

In this section, we'll take a closer look at each method and explore its strengths and weaknesses to help you determine which approach is best suited for your particular use case.

	Rendering from a Template	Rendering from a URL
Advantages	- Fine-grained control over the content and layout of the PDF document - Can easily incorporate dynamic data and logic - Easier to manage dependencies, such as stylesheets and fonts - Can be faster and more efficient than rendering from a URL	- Allows for generating PDFs from existing web pages, such as product pages, blog posts, or help documentation - Supports rendering of single-page applications (SPAs) and other JavaScript-heavy websites - Can print any page without specifying the data needed for that page - Grover takes care of all the rendering for you - Print pages outside the application
Limitations	- Requires development and maintenance of the template - Can be less flexible and adaptable than rendering from a URL - Need to prepare the data required for the rendering template	- Requires both Puppeteer and Chromium to be installed on the server - Requires passing user's cookies or session data for authorization - May not work if cookies have expired or the user is not authenticated - May require additional configuration to ensure security and reliability

I encountered some significant issues when implementing this feature, please take a look to consider:

No-sandbox Puppeteer

When using Grover with AWS, there may be a need to run Puppeteer in no-sandbox mode, which can cause security issues. The no-sandbox mode disables the sandboxing mechanism of Chromium that isolates the browser from the operating system, which means that malicious code can potentially harm the system.

So we need to create a dedicated user for export purposes only, but this can be difficult to configure when switching to this user. This approach involves creating a new user on the server with restricted permissions and allowing only the necessary access to perform the export task. This can ensure better security for the system, but it requires careful configuration and maintenance.

Authorization with cookies

This is because some websites use cookies to store session data or other information, which is necessary for rendering certain pages correctly. Without these cookies, Grover may not be able to access the necessary resources or data, which can result in incomplete or inaccurate PDF output. (The user will receive a PDF file about the login page instead of what they actually need).

When rendering pages synchronously, passing the user's cookies to Grover is usually straightforward. However, when pushing the job to a worker, it may be more challenging to pass the cookies along with the request. This is because workers typically operate in a separate process or thread from the main application, which may not have access to the user's cookies or session data. Rather than that, what happens if the cookie expired at the time the job is executing, how can we retry this job without s newly created cookie?

Render as a background job

To achieve better performance and scalability, it may be necessary to render the PDF as a background job. This is especially true when dealing with many PDF files within a zip file. However, this requires handling the job as a service rather than within the controller, as it is an asynchronous service.

Rendering from a URL

One of the challenges of using the URL to render the PDF is ensuring that cookies still work when executing the job, no one can ensure that, maybe the user logout the account before the job execution or the cookies has been expired.

It’s hard to resolve this problem, we can have some tries to work around this problem, we take it into account but it seems these solutions will do harm to our application

Customize some middleware to completely allow access when exporting PDF
Create a super user that has privileges to access all pages

Rendering from a template

Rendering from a template can be a good alternative for rendering PDFs as a background job, as it doesn't have any significant issues. However, one remaining challenge is that the controller's helper methods may be missing when rendering from a template, and this needs to be done manually.

To overcome this challenge, we can use the view_context and include the required helpers to access them for rendering the PDF. However, this method can be cumbersome and complicated.

view = ActionView::Base.new(EwaPayroll: :PayrollsController.view_paths)

view.class_eval do
    include Rails.application.routes.url_helpers
    include PayrollHelper
    include FontAwesome::Rails::IconHelper
    include Pundit
end

Fortunately, Rails has the built-in feature called ActionController::Renderer can be an efficient and scalable solution for rendering PDFs as a background job. With this feature, arbitrary templates can be rendered without being in controller actions or helper concerns. This eliminates the complexity of manually handling the controller's context and helper methods.

To use ActionController::Renderer, simply call the render method on the renderer object, passing in the desired template name. For example:

PayrollsController.renderer.render(template: '...')

In combination with the Grover gem in Rails, this approach can provide a solid solution for rendering PDFs in the background.