DEV Community: Daniel

Optimize Database Performance in Ruby on Rails and ActiveRecord

Daniel — Wed, 20 Nov 2024 13:00:00 +0000

In Rails, we're more likely to use SQL databases than other frameworks. Unlike NoSQL databases, which can be scaled horizontally with relative ease, SQL databases like PostgreSQL or MySQL are much less amenable to easy scaling.

As a result, our database usually becomes the primary bottleneck as our business grows. Although SQL databases are very efficient, as our growing customer base puts an increasing load on our servers, we begin scaling our instance counts, workers, etc. But we can't just make copies of our database for each new server we spin up. This makes optimizing database performance critical for any serious Rails project.

In this post, we'll explore strategies for optimizing performance to minimize the load on our database. We'll start with some more basic topics like eager loading and the N+1 query problem, database indexing, select, pluck, and immediate loading. Then, we'll move on to more involved topics like performance profiling, scaling via techniques like database sharding, and background jobs using read replicas.

Let's get going!

Getting ActiveRecord N+1s Out of the Way

No discussion of database performance involving ActiveRecord (or any ORM) is complete without addressing the infamous N+1 problem. While most readers are likely familiar with it, it's still worth revisiting, as N+1 queries can (and do) creep into our projects over time, especially as our codebase evolves and we add or update features.

The N+1 problem generally occurs when iterating through a (potentially large) group of retrieved records. Because we haven't loaded their association/s, each loop means an additional query for each associated model we access inside the loop. This can quickly spiral out of control, resulting in a staggering number of queries and causing serious consequences such as crashed pages, an exhausted database connection pool, and memory running out, ultimately grinding our site to a halt.

There are a few useful tools at your disposal to help identify and resolve N+1 problems. The first is simply looking at your server output; generally, this works pretty well, as N+1s are easy to spot. For a more assisted approach, the Bullet gem is a popular tool that automatically detects N+1s in applications and suggests ways to fix them. Another, arguably better option is prosopite, a less well-known option that generally provides better results with fewer false positives (and false negatives).

Eager loading should be used with care, however; while you're probably safe loading has_one associations, has_many can sometimes be dangerous: what happens when we've got a query of multiple joined, many-to-many tables, and eager load one or more of the many-to-many associations?

We can end up loading way too many records into memory. This can crash our app just as surely as a bad N+1. If you're at this point, and both the N+1 and eager loading approaches are causing you problems, it may be time to reevaluate the query itself. Maybe you're trying to load associations you just want to COUNT (which is something you might be able to build into your query), tighten your pagination limits, or consider offloading the query to a background job if possible (which we'll cover).

Database Performance in Rails: Considerations and Optimization

Sometimes, breaking queries down into a couple of steps can both improve performance and simplify a query. Ever find yourself trying to get a subset of records to complete a mind-bending query? In certain cases, just finding the IDs of the subset you want and then feeding them into a second, less complex query can help reduce the need for complex joins, subqueries, etc.

Have you ever fallen victim to Rails' (usually beneficial) default lazy-loading and evaluation, and wished you could load something immediately rather than the first time it's used? Sometimes we need not only the results of a query, but another aspect of it too, like the number of records we've retrieved. Our seemingly harmless code looks like this:

@users = User.where("email ILIKE ?", search)
@total_users = @users.size

Normally, this is fine, but what if we need to count users before we do anything with them (maybe we're displaying the number of users returned in our live search at the top of the page, followed by the users themselves)? We might add COUNT to our query, but then we need to retrieve the records again.

We can get around this by explicitly calling load on our query — for example:

@users = User.where("email ILIKE ?", search).load
@total_users = @users.size

This simple change can save us a query. With our users already loaded, our display of @total_users above the user list will no longer trigger an expensive COUNT followed by another query to get users.

In the other direction, we have load_async with Rails 7. This allows us to asynchronously query the database with multiple requests at once, rather than being locked to one synchronous query at a time. This can be a total game-changer if you're in a situation that can benefit from it. Be careful though, as this (probably unsurprisingly) uses Ruby threads under the hood, so comes with all the same issues you'd expect from threads. That's not to mention the fact that it (potentially) opens our database up to a greatly increased number of connections, and can exhaust our connection pool if we aren't careful.

A large part of database performance in Rails comes down to how you utilize ActiveRecord. While AR is an invaluable tool, it can also lead to performance issues if not used with care. As you're probably aware, it loads every column of every table involved in a query (SELECT *) by default, regardless of how much of that data we actually use. It's pretty easy to forget this though, especially as our queries evolve, and we can end up loading a lot of unnecessary data. It's important to consider whether or not you're really using all of the columns you're loading, or if a select (or even a pluck) statement might be a better fit for your use case.

You're probably familiar with eager loading in Rails, but one thing that's often overlooked is that Rails makes it easy to introduce optional eager-loading (and WHERE clauses, and SELECTs). If you've ever had slightly different requirements for two very similar queries, you might feel like you're left with two options: write two queries, or write covering both use cases, each retrieving more than you need. This might happen in a before_action, where you typically want the same thing for each action in a particular controller. But one controller might benefit from eager loading, more careful column selection, or a search results page which is just filtering the usual index based on some criteria.

There's another way, though:

class User
  def search_user_posts(before_date: nil, optional_selects: [], optional_eager_loads: [], exclude_inactive: false, search: false)
    optional_before_date_constraint = ["created_at < ?", before_date] if before_date
    optional_joins = {user_posts: [posts: :comments]} if search

    self.posts
        .includes(optional_eager_loads)
        .joins(:shared_joins)
        .joins(optional_joins)
        .select(:columns_shared_by_queries)
        .select(optional_selects)
        .where("your shared WHERE constraints")
        .where(optional_before_date_constraint)
  end
end

Rails is smart enough to know not to execute anything here if there's nothing passed in; where(nil) will behave as though it doesn't exist, as will includes([]), etc. This can really clean up your code and help you tailor performance to your needs.

Finally, rethinking your JOINs and WHEREs can sometimes be beneficial. Are you querying a very large dataset where using a JOIN could significantly reduce the initial result set? Try it out, and don't be afraid to run #explain on your query. It can help you determine if switching to a JOIN might work better and identify any missing indexes on critical queries.

Database Indexing

One of our most important (and nearly universally necessary) tools for database performance is indexing. As you're probably aware, indexes are special data structures (typically B-trees) that a database uses to quickly find records, improving retrieval times from O(n) to O(log(n)). However, it's important to add indexes carefully, as indexing columns unnecessarily can actually harm performance.

The trick is to identify the columns and associations that your application needs to query frequently, and then create indexes on those. This allows the database to quickly locate the relevant data without having to scan the entire table.

You can also use partial indexing. Partial indexes allow you to index a subset of a table's rows based on a specified condition. This is particularly useful when your queries frequently target only a specific portion of the data.

For example, maybe your app has a core of frequent visitors, the majority of whom have their own accounts. They make up the lion's share of your traffic, but represent a small minority of your total users. You could define a partial index on the users table like this:

class AddPartialIndexToUsersOnGuest < ActiveRecord::Migration[7.1]
  def change
    add_index :users, :guest, where: "(guest = false)" # the `where: condition` here makes this a partial index.
  end
end

Here, we've specifically targeted rows where guest is false. This optimizes the performance of any query searching for non-guest users by reducing the total rows covered by the index.

It's important to revisit your indexing profile and database schema periodically. The specific indexing needs of your application may change over time as the user base, codebase, and feature set evolve. Doing so often means you end up reviewing your indexes during comfortable periods where you have time to analyze and think about your database, rather than in a panic when an influx of traffic crashes your app.

All of this brings us to our next topic: performance profiling.

Performance Profiling for Your Ruby on Rails App

So, we've identified some major categories of performance optimization; what now? Do we go through our entire app from top to bottom, optimizing line by line? Probably not.

It's pretty unlikely that you have evenly distributed traffic across your whole site. It's even less likely that you can afford to go through your entire codebase; premature optimization may not actually be the root of all evil, but you also don't have time for it. So how do we know what to focus on?

This is where profiling tools come in. While your local server output can provide valuable performance insights into response times, memory use, and query performance, you are only one user. Your local database likely doesn't reflect the scale or patterns of your production database. Valuable as it is for spotting potential performance issues, your local environment doesn't provide the context you need to make decisions about what genuinely needs your attention.

Application Performance Monitoring tools (APMs) like AppSignal for Ruby are essential for monitoring and maintaining our site's performance. They provide comprehensive insights not just at the database layer, but across the rest of the stack, allowing us to pinpoint specific areas (pages, endpoints, and even specific queries) that are causing headaches. With a good APM tool, we can track the performance of any request over various timescales, monitor background jobs, spot memory leaks, set up custom error and performance alerts, and more.

APMs provide visualization tools that make it much easier to spot and understand complex performance trends, leading to quicker, more informed decisions. They help prioritize optimization efforts by highlighting not only the slowest queries and most resource-intensive parts of an application, but also the endpoints that consume the most total time (total request count * average response time). This means we can focus on what's actually important. Remember that page we were worried about that takes a couple of seconds to load? Turns out, its total impact on our site amounts to practically nothing. Our products page though, clocking in at a reasonable 150ms? That's taking up 15% of our total server time.

To find out more about measuring database performance using AppSignal, check out our post Monitor the Performance of Your Ruby on Rails Application Using AppSignal.

Background Jobs

It isn't always an option, but running queries in background jobs is a powerful tool for improving performance.

You might be asking yourself how this could be, given that we only have one database; what difference does it make if we're merely calling on it from another thread or process? But there are a few main ways background jobs can come to our database's rescue.

First, just by nature of the fact that our background jobs run asynchronously, we're pretty free to run relatively resource-intensive tasks — like setting up bulk inserts (or upserts), for example (note: neither validations nor callbacks are triggered for bulk operations) — and potentially distributing large numbers of queries across periods of time (perhaps especially for write operations), possibly with the use of find_each.

You're likely used to using a job to process a CSV or spreadsheet; those tasks tend to lend themselves to asynchronous processing as well as bulk insertions. If you're doing this on your server's main thread, you're likely doing something wrong. But sometimes other, less obvious things we're currently doing on our main thread can be moved to a background job to improve performance on the main thread and reduce load on the database.

Consider a scenario where your site allows people to make potentially large purchases of relatively inexpensive items. A school, for example, might order tens of thousands of pencils at once, and maybe your system creates a PurchaseItem for every single item purchased. Instead of creating or updating thousands of rows one at a time, we can set up a bulk insert in a background job, create all the individual hashes inside a loop, and use insert_all to get the job done — taking advantage of our background worker to set up our bulk inserts, and then writing to the database once, instead of locking it with thousands of writes.

Read Replicas

Background jobs are great, but eventually, even if we're spacing out those large, numerous database queries, we hit a point where we're asking too much of our database. We're probably okay in terms of write performance (we tend to read from our database far more than we write to it, after all), but we're constantly bombarding it with queries. Our site's response times are steadily rising, and user experience is suffering.

At this point, the next logical step may be to introduce read replicas to offload work from the primary database. Read replicas allow you to distribute the read query load across one or more replicas of your database. This can improve performance across the board, as it greatly reduces the number of reads (and thus the total I/O) of your primary database, leaving it less swamped by requests (most of which are likely reads in the first place).

Ideally, large database operations are performed in background jobs reading from replicas, setting up a bulk insert or upsert, and then making one atomic write to your main database. This means you can potentially raise not only the read performance of your site, but can also boost write performance to some degree, as your primary database is consistently freer to accept write requests.

While read replicas are not painless to set up, they can provide the extra performance our site needs.

Database Sharding At A High Level

Remember how we said SQL databases don't scale horizontally? Technically, they can scale horizontally through database sharding. Sharding involves splitting up your database into multiple databases, which are then distributed across multiple servers (and often across geographical locations).

There are several approaches for doing this, including:

Geographical sharding (based on user locations).
Vertical sharding (splitting into different tables or groups of tables based on access patterns, with each shard storing a different subset of the database’s tables).
Functional sharding (splitting based on business function; like separating out billing from user content).

And more. Each approach has its use cases, and some naturally lend themselves better to some businesses than others.

As you've doubtlessly realized by now, sharding is generally difficult, and depending on your business, sometimes not even possible. It adds complexity, developmental and maintenance overhead to our apps, and potentially a lot of it. The good news is that if you don't have the resources to cover this, you probably don't need it in the first place. The bad news is that even if you do, it still complicates things and means increased costs (presumably balanced by the potential benefits).

But if you've done all you can with query optimization, read replicas, and caching, sharding might just be the next necessary step — and one that can greatly increase your I/O capability.

Words of Caution and Final Words: Database Optimization in Ruby on Rails

We need to be conscientious in how we approach our optimizations. It's possible to over-index (or index the wrong things, which hurts write performance). In the same way that N+1s can get out of control if we're not careful, so can eager loading, and it's not always immediately obvious when we're developing locally (hence the importance of at least approximating your real-world site locally).

Keep an eye on things via an APM, as traffic and user patterns can change and new ones can emerge, not just over time but potentially week to week and season to season, depending on your business.

There's no one-size-fits-all approach, so we have to be mindful of how we tailor our optimizations to our own site. Remember not to waste too much time trying to optimize things until you have an idea of the impact they'll have on your site.

That doesn't mean you shouldn't try to write generally efficient code, just that there's generally no reason to agonize over tweaking things to be absolutely optimal or wasting time on micro-optimizations. Try to keep the time-to-benefit ratio in mind, and how what you're working on will likely fit into your existing site as it is now, and as it evolves.

If you're interested in diving deeper into aspects of database optimization like sharding, you can learn more from the Rails guides. Also, check out Paweł Urbanek's great article on load_async.

Happy optimizing!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Expressive Ruby and Rails: Communicate Effectively With Your Code

Daniel — Wed, 18 Oct 2023 14:38:36 +0000

Ruby is an expressive language. This is no accident; Matz very consciously designed Ruby as an intuitive language to more or less read like English. It's safe to say that he succeeded. Methods are named very carefully, and do what they say they do; they also tend to have inverse methods which do the opposite.

In this post, we'll look at why expressive code is important and its impact on your productivity as a developer. Then, we'll explore how to best use some of Ruby's methods.

Let's get started!

The Importance of Expressiveness in Ruby

Expressive code goes beyond readability ("What does this code do?"), and clearly communicates intent: it clues us into the developer's considerations and concerns, and the edge cases they were accounting for. It makes clear not just what the code does, but why it was written how it was.

Why is this important? It saves us from spending a lot of time familiarizing ourselves with the context of the code we're working on, clues us in to what we should be looking out for when we modify it, and keeps us from potentially bulldozing carefully-coded functionality and measures meant to safeguard our application.

Ruby takes to heart the principle that good code should be self-documenting; it largely obviates the need for comments. This is likely one of the reasons we often associate commented code in Ruby with bad code: because, in many cases, it is. Working with external APIs or unfamiliar gems is one thing, but generally, for most of our core functionality, writing idiomatic Ruby means we don't need to write (or read) comments. Usually, any time I find myself with comments all over the place, it's a sign I need to refactor.

Ultimately, expressive code saves us time, and our company money.

The Impact of Expressive Coding on Productivity

What problems do we face if we aren't conscientious about how we write and structure our code? The answer is at once straightforward and easy to forget in the moment. The most obvious issue — a lack of clarity — in addition to being its own problem, can lead to a host of other headaches:

Opacity

In addition to being confusing, inexpressive code also turns the thought process of the developer who coded it into something of a black box, rather than a window into their decision-making.

As touched on above, this can force the developer maintaining our code to delve into the implementation in greater detail than they should to get context. This only serves to derail us from our task at hand, and makes our lives harder.

Loss of Trust

When a particular feature or area lacks expressiveness, it can be frustrating, but it is also an opportunity to improve our app. But when a lack of expressiveness is a pattern across our entire codebase, we lose faith in our code as a reliable source of truth to communicate the potential values it may be handling.

This inevitably leads to constant defensive programming, which can serve to further muddy the waters. We may wind up accounting for potential values that aren't even real possibilities, magnifying the existing problem and further eroding trust. In addition, we might add verbosity to our code, and send potentially misleading signals to other developers (and even our future selves) about what we're accounting for and trying to accomplish.

On the other hand, a lack of trust can lead to bad assumptions when the right method is actually being used. Consider a scenario where someone has just spent time cleaning up a section of code and spots something like if @user.address.present?.

Rather than stopping and thinking: "hey, there must be a reason present? was used," they might think: "hey, this particular field is optionally set by the user... so either they set it and it's a string, or it wasn't set, so it's nil." So they change it to if @user.address. Except, what happens when the user has a value set, and then later removes it and submits the form? We'll now have an empty string (which now, without present?, evaluates as truthy).

Type Issues

Constantly having to check for potential data types and the possibility of nil values can be genuinely exhausting and really slow us down. It not only wastes time by forcing us to work out the possible data types of inputs, but it can also break our focus and flow, and add unnecessary cognitive overhead.

In some ways, we can think of Ruby's expressiveness as an answer to a lack of static typing. By employing clear variable names and self-explanatory methods that do what they say in a fixed, predictable manner, we can automatically infer the type (or potential types) of data we're working on in any given line with little effort.

Maintenance and Extensibility Issues

Inexpressive code can be a nightmare to maintain, and it often leads to tangled code, quickly becoming tough to modify without breaking something. It's even harder to extend with new features and functionality, and can become a bug-prone black box that's slow and challenging to develop for.

Hurting the Bottom Line

We pile up technical debt, which costs us increasingly painful amounts of time, loses our business money, and potentially hurts our reputation if this leads to missed deadlines or buggy releases.

On top of all this, the less expressive our code is, the more painful turnover becomes; onboarding new developers takes longer, as it will be a while before they're productive and know the codebase, and losing developers who do know it has a greater impact. And, of course, even when they're up and running, they and everyone else working on the application will simply be less productive than they could be.

Now, let's focus on how we should best use methods in Ruby and Rails to avoid some of these issues.

Comparing Similar Ruby and Rails Methods

As we touched on early in this post, Ruby and Rails come together to offer us a wide variety of methods that accomplish an array of everyday tasks and answer basic questions we might need to ask as programmers. At first glance, there might appear to be a lot of overlap, and we may think that many of these methods can simply be used interchangeably without issue.

Of course, as we know, these methods were named very carefully. Both Ruby and Rails were designed with close attention to what these methods do and don't do.

As a result, methods (excluding aliases) each behave slightly differently, with different intended purposes. Their correct use can help us provide specific context to clue in anyone reading our code to what's going on. Conversely, failing to do so can have the opposite effect. Let's take a look at some methods that sometimes get used interchangeably, to our detriment — starting with nil?, empty?, blank?, and none?.

`nil?`

This method does exactly what you'd expect: it tells us whether a value is nil. I generally like to use it to avoid doing things like !@user, but it can also indicate that a value is not a boolean, if that's not already clear. While it's difficult to misuse, it's easy to use another, technically valid but misleading method (such as blank?) in its place. Don't do this!

If a value can only be nil or truthy, nil? should always be used. Doing otherwise can mislead anyone modifying the code and potentially lead them to code for non-existent edge cases, which is both a waste of time, and adds unnecessary complexity and verbosity.

`empty?` (Nearly Opposite Method: `any?`)

empty? is useful when you want to check whether a collection array, string, hash, or other type of collection contains no elements. It's a straightforward and intuitive method that returns true if the collection contains no elements, and false otherwise.

For example, "".empty? and [].empty? both return true, because the string and the array do not contain any elements. Similarly, {}.empty? returns true because the hash does not contain any key-value pairs.

When you use empty?, you're communicating to other developers that the object you're checking should be a string or collection of some sort, and you're interested in whether it contains anything.

`blank?` (Opposite Method: `present?`)

blank? is often used to check if a string is empty. However, as you're likely aware, it checks for more than just empty strings; it will return true if a value is nil, false, or an empty array or hash. Essentially, it asks and answers !object || object.empty?, so we expect that the potential values are probably either nil, or an empty string, array, or hash — this is what you're communicating to other developers when you use this.

It is often used in place of empty? or nil? when it really shouldn't be.

`none?` (Opposite Method: `all?`)

This asks whether an array or hash is empty or only contains falsy values, and it's not too often misused. Sometimes, though, we might see someone use blank? instead.

This is a mistake. In addition to the fact that blank? asks: "is this object falsy or empty?", none? asks not only: "is this array or hash empty?", but also: "...or does this array have no truthy values?". [false, nil].blank?, for example, will return false, but [false, nil].none? will return true (likewise, any? would return false).

So not only have we introduced ambiguity, we've introduced a bug waiting to happen — and sadly, this one is not a feature.

`size` Vs. `length`

`size`

Put simply: size is for collections (ActiveRecord Collections, arrays), and should almost always be used for them. It never makes sense to call on a string.

`length`

length is generally for strings, and except in specific cases, shouldn't be used for collections (and probably never for arrays).

`boolean?` Methods in Ruby

Every Ruby boolean? method answers a yes or no question: "Is this value truthy or not?". Most Rails developers are aware of this and generally use the question mark to good effect, adding it to the end of the boolean methods that they write. But many may not be aware that Rails itself does, too.

Any Rails object automatically responds to a column with a ? on the end. So, a verified column on your user table can be called with @user.verified?. Setting up a standard for your team to only use these methods for boolean columns can help provide even more context for other (especially new) team members. They won't have to check the schema to see if verified is maybe a timestamp or even a string rather than a boolean.

And let's not forget the methods we write ourselves; generally speaking, any method we write that answers a yes-or-no question should return a boolean method, and end with a question mark.

For example, say we have a method that tells us whether or not a value is outside a particular limit. We can define something like this:

class Vendor
  def use_exceeds_limit?
    get_time_slot_use_count(1.month) > time_slot_tier.limit
  end

We can then conditionally email to let users who have exceeded the limit know they'll be charged at a higher rate for the month:

class DailyLimitCheckJob < ApplicationJob
  def perform(vendor)
    VendorMailer.alert_vendor_limit_exceeded(vendor) if vendor.use_exceeds_limit?
  end
end

So, great, we know that methods that end in a ? return a boolean. But what about other methods?

Note the method get_time_slot_use_count. It may be slightly more explicit than it needs to be, but it also tells us exactly what we're getting: an Integer. If it were get_time_slot_usage, we might wonder if it wasn't a Float, or even an ActiveSupport::Duration (with a return value like 173.hours). Being explicit helps us know what we'll get without looking at the actual implementation elsewhere in the Vendor class.

So, we can also apply this principle elsewhere: we probably don't need it for strings like name or title, but if we're assigning variables to dynamically hold parts of a query, we probably do want to name it something like query_string, rather than query.

The same is true of arrays and hashes: generally, a plural name (say, products) indicates that we're dealing with a collection. We often don't need to explicitly specify "hash", either; consider pagination_options. We know options in contexts like these are typically hashes. But sometimes we have products as a key-value pair collection; we should probably name this products_hash.

Further Resources

For those seeking a deeper dive into the world of Ruby, a number of resources are available.

Check out the books The Ruby Programming Language and Ruby Best Practices, the official Ruby docs, and Rails Guides.

Wrapping Up

In this post, we've walked through the pitfalls of inexpressive code, including a lack of clarity, a decrease in trust, and other problems. In contrast, well-crafted, expressive code can help avoid issues like technical debt, make our lives easier, and save money.

The design philosophy of Ruby prioritizes expressiveness and intuitiveness. Properly leveraging the tools Ruby provides to communicate your intent can eliminate the need for extensive comments and lead to substantial savings in both time and resources.

To fully unlock the power Ruby and Rails affords us, it's critical to be conscientious in your variable and method naming, and to use the right method for the job. Expressive code makes you — and everyone around you — more productive.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Metaprogramming in Ruby

Daniel — Wed, 02 Aug 2023 14:29:15 +0000

You've heard of metaprogramming: code you write that generates other code dynamically. Without it, Rails as we know it wouldn't (and couldn't) exist. But there's a good chance you've never done it yourself, and it's not hard to see why; even a brief excursion into the realm of metaprogramming can leave you beset with strange and foreign methods, unfamiliar syntax, and downright mystifying blocks of code.

It's true: metaprogramming is a relatively advanced topic. If you want to really dig in and leverage it on a deep level, it will take time, effort, and a different way of thinking than you're used to. But there's good news: You don't need to wade too deeply into the metaprogramming waters to discover useful methods and techniques to make your life and workflow a little easier.

In this post, we'll take a look at metaprogramming methods like send, define_method, and method_missing and show how they can solve problems we sometimes run into, even in normal Rails applications. But first, let's briefly define metaprogramming and explore when it might come in handy.

When Is Metaprogramming Useful?

Maybe you've got a couple of models in your app that, while being very similar, aren't identical — they have different attributes and methods. But both need to be acted on in a job, and you don't want to write what is essentially the same code more than once.

Maybe once (or twice), you hacked together a ~~kludge~~ masterpiece for a tight deadline on a new feature that got the job done, but you modified an existing model rather than creating a new one the way you would have if you'd had more time. Now that feature needs to be expanded, requiring you to do it the "right" way. This can be intimidating, especially if your codebase references your current implementation all over your app.

Or maybe you rely on a third-party gem that isn't well-maintained, and you discover far too late that it's got a bug in it — right in the middle of a line of code full of metaprogramming.

I've had all these things happen, and I solved them by leveraging some light metaprogramming, all without having to be an expert, and without too much effort. You can, too.

What Is Metaprogramming in Ruby?

Metaprogramming is a set of techniques that Ruby offers us to write code that dynamically writes other code for us. Rather than working on data, metaprogramming works on other code. This is great because it means we can write more dynamic, flexible, and adaptable code if the situation calls for it.

It can help DRY up portions of our code, dynamically define methods we might need, and write useful and reusable pieces of software (like gems!) to make our lives easier.

In fact, Rails leverages metaprogramming on a large scale and to great effect. Any time anyone talks about Rails' magic, they're really talking about its use of metaprogramming.

Caveats

Despite how useful metaprogramming is, it isn't without its drawbacks.

One issue is readability: a little metaprogramming here and there can go a long way and likely won't cause many headaches, but lots of it will hurt the readability of your code.
Many Rails engineers probably are not very familiar with it, so relying on it too heavily can prove frustrating both to your future self and to others who need to work on your code.

Maintainability is also a concern; heavy use of metaprogramming can be mind-bending for even experienced Ruby developers. This means you should document your code. Ruby is an incredibly expressive, intuitive, and readable language, but if you're doing things others might find confusing, you should leave comments. You should also use access modifiers - e.g., private, protected - the way you would with other methods.

Another potential source of trouble: monkeypatching. While monkeypatching (dynamic modification of a class — typically, a Rails core class) can be useful, it should also be used sparingly and carefully. If we're not mindful, we can modify behavior in ways that, while perhaps solving one of our problems, can easily create dozens of others. For example, rather than fixing a small bug, adding a new, uniquely-named method, or simply extending the capabilities of a class, we might modify the behavior of existing methods used not just by our own application but by our gems (and even Rails itself), with potentially disastrous consequences. Check out our post Responsible Monkeypatching in Ruby for more on monkeypatching.

Lastly, metaprogramming can make it hard to find things you're looking for, especially if you didn't write the code in the first place. If you find yourself trying to figure out how it's possible CompanySpecificClass#our_special_method exists on an object but isn't defined anywhere, metaprogramming could be the culprit (in this case, probably the use of define_method).

Basic Ruby Metaprogramming Techniques

Let's explore some of the methods we mentioned earlier: send, define_method, and method_missing.

First, we'll take a look at send, what it does, and how it can help us DRY up our code.

Using `send`

Remember the example we mentioned earlier (under the header 'When Is Metaprogramming Useful?') of a couple of similar models with different attributes and methods? In this case, both models need to be acted on in a job in a more or less identical manner.

Essentially, invoking .send allows us to dynamically pass methods (along with arguments) to an object, without necessarily knowing what that object (or method) might be at the time we write the code.

First, a word of caution: send can actually call both public and private methods. If you want to be very explicit (and careful), you can use public_send to ensure it's clear you're calling a public method. I've never done this, but I've also never used it to call any private or protected methods; if you do, you might want to use public_send and send for each use case.

So how does it work? Pretty simply. You just pass the method's name to the object you want to call like this: my_object.send(:method_name). If you need to pass parameters/arguments, you can do so: my_object.send(:method_name, argument1, argument2...). send accepts parameters the same way other methods do, so you can use named arguments, too.

So rather than doing something verbose like this:

def sell_or_refund_or_return_item(item, action)
  if action == "sell"
    item.sell
  elsif action == "refund"
    item.refund
  elsif action == "void"
    item.void
  elsif action == "return"
    item.return
  end
end

We can simply pass our method as an argument and call it with send:

def process_item(item, action)
  return unless item.respond_to?(action)

  item.send(action)
end

Note: For the sake of simplicity and brevity, we're not raising an error if our object doesn't respond_to? the action passed into our above methods. But in a real app, we'd probably want to raise an error to bring our attention to the fact something in our codebase is calling a non-existent method/attribute on the object.

`send`: An Example Scenario

Let's consider the following example. We've got a Purchase model, representing a given purchase from an online store. That purchase could be anything from a single item of one type to many different items.

In the instance below, we're interested in people who bought a particular subscription, which may contain tickets to events (purchase_events), and/or vouchers for products (purchase_products).

Products and events are different things, but there's a lot of overlap, like price, status (sold, returned), etc. In this example, we will refund all instances of a particular item from a subscription, because customers were unable to redeem their purchase (an event was rained off, a company we source from ran out of a collectible, etc.).

class RefundAllInstancesOfItemJob
  def perform(item)
    purchases = Purchase.joins(:subscriptions)
                        .includes(:purchase_events, :purchase_products)
                        .where(subscriptions: { id: item.subscription_id, status: "active" })

    association, foreign_key, foreign_key_value = item.is_a?(PurchaseEvent) ? [:purchase_events, :event_id, item.event_id] : [:purchase_products, :product_id, item.product_id]

    purchases.find_each do |purchase|
      purchase.send(association)
              .where(purchase_id: purchase.id)
              .where(foreign_key => foreign_key_value)
              .each { |item| item.update(status: "refunded") }
    end
  end
end

What have we done here? Ordinarily, we might have ended up writing some if/else logic and duplicating most of this code or even two separate jobs. Instead, we've dynamically sent the appropriate association and foreign key to objects within our query, keeping our code DRY.

We've laid out the keys and associations here explicitly, but they could be passed in as arguments if we set up our call to the job differently. Our customers have now been refunded for the particular item(s) in this subscription cycle! (Let's assume there's a callback on those models that refunds the user when the item is marked "refunded").

Using `define_method`

Now, let's look at define_method and how it can help us with our earlier example where we moved over from storing data on one model where it didn't really belong to a new dedicated model.

Let's say we currently have a Presentation model. When we first added video capabilities to our app, we did it because our biggest customer had to play a single, long video about their business to their shareholders. We had a tight deadline, so rather than create a new, dedicated model, we just tacked on a few columns to our Presentation table, and it got the job done.

Now, though, because we've advertised our app's recording and streaming capabilities, other clients are interested — and they need more than one video per Presentation. Unfortunately, we're still in a bit of a time crunch and have to roll this out fast.

We decide the quickest way to accomplish this without changing tons of code is to:

Create a new VideoAsset model
Move existing columns over from Presentation
Set a boolean current_asset which will update when our clients select the video they want

class Presentation
  has_many :video_assets

  def current_video_asset
    video_assets.find_by(current_asset: true)
  end

  # Replace our dropped columns as new instance methods on Presentation
  # We've used the safety operator (&) on send the way we would any other method with a potential nil return value
  ["status", "playback_id", "asset_id"].each do |column_suffix|
    define_method(column_suffix) do
      current_video_asset&.send(column_suffix)
    end
  end
end

What's going on here? We're dynamically defining methods on our Presentation model to replace the columns we dropped and moved over to VideoAsset. The existing calls in our codebase to @presentation objects will now first find the current VideoAsset associated with our Presentation and then call the corresponding column. We don't have to go around updating controllers and views everywhere!

It's worth noting that the methods defined above could also have been accomplished using ActiveSupport's delegate helper. delegate allows you to use this pattern more often, abstracting away the metaprogramming implementation:

class Presentation
  has_many :video_assets
  delegate :status, :playback_id, :asset_id, to: :current_video_asset

  def current_video_asset
    video_assets.find_by(current_asset: true)
  end
end

Using `method_missing`

Finally, let's look at method_missing. It does pretty much what you'd expect, given its name — it allows you to account for methods that don't exist but are called on an object or class. Let's take a look at turning methods we've placed in our classes into methods that test for truthiness.

class User
  def method_missing(method_name, *args)
    if method_name.ends_with?("?")
      regular_method = method_name.to_s.sub("?", "")

      result = self.send(regular_method, *args)

      result.present? && result != 0
    else
      super
    end
  end
end

So, what's happening here?

Well, we've basically recreated an ActiveRecord feature for our User model. Any column that exists on a table in Rails has an identically-named method ending in a ? added to instances of its corresponding class. We've done something similar with our User class — any undefined instance method called on a user object ending in a ? will be tried against a method of the same name without its question mark, and turn the result into a boolean.

So, for example, @user.purchase_totals? will (rather than return a decimal representing the total amount of money a user has spent in our app) simply return true if the number is nonzero — otherwise, false.

Note the use of present? here. It accounts for things like empty strings and arrays, which otherwise would return true if we'd used a double-bang to assess if it was truthy or not.

If there's no match, we default to calling super, resulting in the expected behavior (a NoMethodError being thrown).

More Advanced Metaprogramming Techniques in Ruby: An Overview

We've covered some useful methods that can be used sparingly to help us out in everyday situations. But what about more advanced uses of metaprogramming? What else is it good for?

DSLs: As mentioned, ActiveRecord, the ORM (and DSL) that ships with Rails heavily leverages metaprogramming. That's where all those automatic methods on our objects come from. Every column we add to a table becomes two methods on our instances — column and column= (and column? for those who were paying attention!). It isn't magic that's making this happen, it's metaprogramming — in fact, the use of the same method_missing method we talked about earlier! Metaprogramming is also responsible for Rails' ability to automatically create methods like find_by_first_name and find_by(first_name: "name").
Gems: If you're building a gem, chances are you'll want it to be flexible, adaptable, and to work with more than just one specific Rails stack. Most of the gems you regularly use have at least some degree of metaprogramming, and many use a lot to achieve the level of flexibility they offer.
Dynamic APIs: Metaprogramming can help us design dynamic APIs by allowing us to define methods on the fly based on runtime information, rather than hardcoding every possible method. For example, a RESTful API might expose resources with dynamic paths and attributes based on the data model of the underlying application (this may sound familiar — Rails' routing system does this). We can generate methods for each resource at runtime based on the database schema and dynamically respond to HTTP requests.
Frameworks: As mentioned, Rails wouldn't be Rails without metaprogramming. While it's a particularly magical framework, almost any framework you build will need plenty of metaprogramming. If you're going to build your own (presumably far more lightweight) framework, you'll need to leverage metaprogramming.

Wrapping Up

We've covered some real ground here! We learned about dynamic method definition with define_method and method_missing, and about dynamic method invocation with send and public_send.

We also talked about metaprogramming's pitfalls, including its issues with readability, maintainability, and searchability. Finally, we touched on more advanced use cases like writing gems, DSLs, and frameworks.

Further Learning

There are a lot of resources out there for taking Ruby metaprogramming further.

RubyMonk offers a short but free course.

A couple of other paid courses include:

Ruby Metaprogramming — Complete Course from Udemy
Chris Oliver's comprehensive Advanced Ruby: Behind the Magic. In addition to digging deep into Rails, this also has a section on metaprogramming and DSLs.

An oft-recommended book is Metaprogramming Ruby 2: Facets of Ruby.

I hope you found this post useful. Thanks for reading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

A Guide to Rails View Helpers

Daniel — Wed, 08 Feb 2023 11:20:40 +0000

Views in Rails don't do much besides showcasing what we want. Sure, they might render slightly different results depending on who's watching (an admin or a logged-in user has a different experience than a guest user, for example), but they don't do much processing on what they're given.

Or at least they shouldn't. Often though, Rails projects wind up with a lot of logic in their views.

In this post, we'll explore how to use Rails helpers to keep our views clean and readable.

But first, let's see why too much logic in our views can become problematic.

The Problem with Logic-heavy Rails Views

Logic-heavy views are bad for a lot of reasons. We might end up with logic-heavy views because of the pressure of a crushing deadline, or they might result from inevitable legacy code buildup. New features get introduced along with new edge cases, and it seems harmless to update <%= @post.name.titlecase %> to <%= @post.name.titlecase.gsub("#", "") %>.

But before you know it, this can snowball, even if you don't keep tacking on little additions to this particular line of code. If you keep doing things like this elsewhere in the view, you end up with an ugly, difficult-to-read mess.

So what do we do? Should we litter our controllers with dozens of instance variables like @normalized_titlecased_post_name = @post.name.titlecase.gsub("#", "")? Bloat our models with tons of methods intended only for use in our views? Probably not.

Enter Rails helpers.

What Are Helpers in Rails?

A helper is a method intended to abstract logic out of our views, leaving them more readable, less cluttered, and not directly responsible for processing what they've been passed by our controllers.

Rails already ships with a lot of useful helpers to begin with. Some you're likely already familiar with, like form_with, link_to, and image_tag, to name a few.

Most of the helpers you write yourself will naturally be much more specific to your app than what Rails includes, but you will likely still find many Rails helpers very useful.

If you haven't already, familiarize yourself with the helpers Rails provides — otherwise, there's a decent chance you'll end up trying to solve a problem that's already been solved.

I once wrote a method early in my Rails career to convert a number to a dollar amount, only to discover later that Rails already had a solution — number_to_currency!

Organizing Your Helpers in Rails

By default, a new Rails application has one helper — ApplicationHelper — which all other helpers should inherit from. You've likely noticed that any time you create a controller via rails g controller, Rails automatically creates a new (pluralized) helper module for you as well.

While this form of organization (having a helper module for each controller) is both useful and important, it's also a bit deceptive. The methods defined in any of these helpers are available in all your views. So if you define my_generic_method() in, say, UsersHelper, and an identically-named my_generic_method() in TransactionsHelper, they won't only be available in their respective views — one will simply overwrite the other and almost certainly break something.

So you should be fairly thoughtful when naming your helper methods. Don't assume there's an invisible User:: namespace or user_ prefix in front of the methods in your UsersHelper, because there isn't, real or implied.

Any helper methods generic enough to be used all over (most of these probably won't be model-specific) can go in your ApplicationHelper. Methods specific to view folders (e.g., /views/users) belong to helpers of the same name.

Again, this is strictly for organizational purposes, given all helper methods are available to all views. But, as you might imagine, it will definitely confuse other developers (and likely your future self) if you end up finding User-specific methods in your ZooAnimalsHelper.

Note: Before Rails 4, this was not the case. You can turn this behavior off by setting config.action_controller.include_all_helpers = false in your application.rb.

Do's and Don'ts for Helpers

Helpers in Controllers

You can include helpers in controllers. In modern Rails versions (5+), you can simply access them in your controllers via helpers (e.g. helpers.number_to_currency).

It's often tempting, and you can absolutely abuse this — I won't pretend I never have. But generally, you shouldn’t. It's okay to do this occasionally, and if you must, do use the helpers method rather than include on the entire helper module in your controller. include dumps every method defined in that helper into the controller, potentially conflicting with your controller methods.

If you keep finding you want to share a helper across different layers of your app, consider whether it might be better off elsewhere, like in a model.

Helpers and Instance Variables

Helpers can access any instance variables available for the view they're called in. The instance variables can be referenced inside your helpers without being passed explicitly as method arguments, but this is not advised.

Instead, pass your instance variables to your helpers as regular arguments. Doing otherwise tightly couples your controllers and views (and helpers), making reuse, refactoring, testing, and debugging difficult.

On rare occasions, you might want to reference instance variables that aren't passed as method arguments inside your helpers. You could have a helper method that's so specific, it only belongs in one spot, or the opposite — it only leverages an instance variable you have everywhere.

This is still somewhat iffy, but ultimately it's up to you. I will say though that if you're not very certain, err on the side of caution. As with anything else, you'll get a feel for things as you see what does and doesn't work over time.

Never Modify Objects with Helpers

This is probably obvious, but a helper should never modify the state of a passed object or (even worse) modify that object in the database. Remember, they're responsible for helping the View display things cleanly, not changing anything.

Never Make Database Calls with Helpers

Helpers should also never be responsible for making calls to the database — no part of the view should. Not only does doing so badly violate the separation of concerns, it's also incredibly confusing to other developers (where is this query coming from?), and can make tracking down problems difficult. I once spent hours completely baffled by where an N+1 was being introduced, only to discover some queries inside a loop embedded in the view.

Don't Let Helpers Become a Dumping Ground

One final word of caution: don't, as is so often the case, let your helpers become a dumping ground for one-off methods and little snippets of code that either serve little purpose, or might be better off as another kind of method. They should be relatively simple methods, and only concerned with the view; business logic really belongs in our models.

Generating HTML with Rails Helpers for Views: An Example

Rails helpers can work with and generate HTML for our views. Remember Rails' built-in helpers? A content_tag will let us generate our own HTML.

Let's say we have a zoo and want to display a list of exhibit times for a given group of animals a customer selects. We might have something like this in our view:

<div>
  <% @animals.map do |animal| %>
    <div class="time-heading">
      <%= animal.name %> (<i><%= animal.species.name %></i>): <%= animal.exhibit_time.in_time_zone(animal.exhibit_time.time_zone).strftime("%m/%d/%Y %Z") %>
    </div>
  <% end %>
</div>

There are a few issues with this. For one, it's a bit hard to read. We're also calling things like in_time_zone and strftime to format and display the exhibit time, along with a lot of method chaining.

It's also in no way reusable. Let's say we need to display this (or something similar) in different places on our site. If we then later decide it needs a style or content update, we have to change it everywhere. This is annoying for obvious reasons (in addition to the fact that it can be tricky to find every example if they're all slightly different).

Let's see what handling this in a helper might look like:

# helpers/animals_helper.rb
module AnimalsHelper
  def display_exhibit_times(animals, dom_class: nil, style: nil)
    exhibit_strings = animals.map { |animal|
      "#{animal.name} (<i>#{animal.species.name}</i>): #{animal.exhibit_time.formatted_start_time}".html_safe
    }

    exhibit_strings.map { |exhibit_string|
      content_tag(:div, "#{exhibit_string}", class: dom_class, style: style)
    }.join
  end
end

Admittedly, this may not be the prettiest example, but it's considerably easier to read and, more importantly, removes the logic from our view entirely. It's also reusable and flexible; notice the two named arguments we've passed in — dom_class and style. Because content_tag allows us to pass in HTML class and style information, we can pass this in any time we call this method to customize it in different places as needed.

Another thing you may have noticed is the transformation of the time formatting into a call from exhibit_time. This is just to point out that not all methods used in the view have to be helpers; this might be a model method, or a mixin shared across models with times and dates for easier formatting. (That isn't to say it might not also make sense to use a helper here instead; it might.)

Note: .html_safe is needed to translate the </i> tag to HTML. .join is required to return multiple content tags for use in the DOM.

Our view now becomes:

<%= display_exhibit_times(@animals, dom_class: "time-heading") %>

Isn't that better?

A Second Example

Let's consider another example. We need to email our customers their tickets, and we'd like to provide them with an online menu for a dining area at our zoo.

In both cases, we'll probably want to render QR codes. Let's use RQRCode, a gem that can do just that for us:

<div>
  <%= RQRCode::QRCode.new(@ticket.qr_code).as_svg(color: "black", shape_rendering: "crispEdges", module_size: 5, standalone: true, use_path: true) %>
</div>

This isn't ideal for a number of reasons. For one thing, we're instantiating objects in our view. For another, we're passing every single required option, when most of them will probably be the same throughout our app.

Let's move this to a helper:

# helpers/application_helper.rb
def render_qr_code(code, options = {})
  qr_code = RQRCode::QRCode.new(code)

  defaults = {
    color: "black",
    shape_rendering: "crispEdges",
    module_size: 3,
    standalone: true,
    use_path: true
  }

  qr_code.as_svg(defaults.merge(options)).html_safe
end

Our view becomes:

<div>
  <%= render_qr_code(@ticket.qr_code, module_size: 5) %>
</div>

Much cleaner, and now we can reuse it for our restaurant!

One More Trick: The `tag` Helper

Do you ever find yourself interpolating conditional class names in your elements?

Say we're listing our animal exhibits on our main page, and want to highlight certain exhibits if they're currently featured:

<div class="border-black margin-standard <%= 'bg-featured text-bold' if @exhibit.featured? %>">
  <%= @exhibit.name %>
</div>

As of Rails 6.1, you can set conditional classes using the tag helper.

<%= tag.div class: ["border-black margin-standard", "bg-featured text-bold": @exhibit.featured?] do %>
  <%= @exhibit.name %>
<% end %>

The classes "bg-featured" and "text-bold" will only be included if it's a featured exhibit. This can make things easier to read, especially if you've got multiple classes with multiple conditions!

There Isn't Always a 'Perfect' Solution

So you've been using helpers for a while now and are comfortable with them. You've refactored your views, pulled methods out of your models and your controllers, and everything's cleaner, easier to read and test. You've noticed you're introducing fewer bugs into your application.

And yet — you still have a bit of logic in some of your views, a few calls to helpers in your controllers, and a few methods that don't belong in the right place.

That's completely fine. This is normal and, at least in my opinion, inevitable. Some devs claim there's always a solution — be it a helper or model method, presenter/decorator class, etc., but personally, I'm not convinced.

Suppose you continually find you want to use a particular method across multiple layers of your app. In that case, it might be a good candidate for a concern, model method, or part of a separate module. But sometimes, there isn't an absolutely perfect solution for every use case or a perfect place to put code. There's certainly no such thing as a 'perfect' app. And that's okay.

Wrapping Up

In this article, we explored how to make your Rails views cleaner and more readable using helpers. We looked at how to organize your helpers, some do's and don'ts, and how to generate HTML for your views using helpers.

We finally argued that even if you master helpers, you'll likely still never achieve the 'perfect' Rails app. However, you'll certainly have an app that's easier to maintain.

I hope you've found this post useful.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

DEV Community: Daniel

Optimize Database Performance in Ruby on Rails and ActiveRecord

Getting ActiveRecord N+1s Out of the Way

Database Performance in Rails: Considerations and Optimization

Database Indexing

Performance Profiling for Your Ruby on Rails App

Background Jobs

Read Replicas

Database Sharding At A High Level

Words of Caution and Final Words: Database Optimization in Ruby on Rails

Expressive Ruby and Rails: Communicate Effectively With Your Code

The Importance of Expressiveness in Ruby

The Impact of Expressive Coding on Productivity

Opacity

Loss of Trust

Type Issues

Maintenance and Extensibility Issues

Hurting the Bottom Line

Comparing Similar Ruby and Rails Methods

nil?

empty? (Nearly Opposite Method: any?)

blank? (Opposite Method: present?)

none? (Opposite Method: all?)

size Vs. length

size

length

boolean? Methods in Ruby

Further Resources

Wrapping Up

An Introduction to Metaprogramming in Ruby

When Is Metaprogramming Useful?

What Is Metaprogramming in Ruby?

Caveats

Basic Ruby Metaprogramming Techniques

Using send

send: An Example Scenario

Using define_method

Using method_missing

More Advanced Metaprogramming Techniques in Ruby: An Overview

Wrapping Up

Further Learning

A Guide to Rails View Helpers

The Problem with Logic-heavy Rails Views

What Are Helpers in Rails?

Organizing Your Helpers in Rails

Do's and Don'ts for Helpers

Helpers in Controllers

Helpers and Instance Variables

Never Modify Objects with Helpers

Never Make Database Calls with Helpers

Don't Let Helpers Become a Dumping Ground

Generating HTML with Rails Helpers for Views: An Example

A Second Example

One More Trick: The tag Helper

There Isn't Always a 'Perfect' Solution

Wrapping Up

`nil?`

`empty?` (Nearly Opposite Method: `any?`)

`blank?` (Opposite Method: `present?`)

`none?` (Opposite Method: `all?`)

`size` Vs. `length`

`size`

`length`

`boolean?` Methods in Ruby

Using `send`

`send`: An Example Scenario

Using `define_method`

Using `method_missing`

One More Trick: The `tag` Helper