Create a Business Language for a Rails Application

As web developers, we tend to approach problems with traditional low-risk solutions. When all you have is a hammer, everything looks like a nail. When you need complex input from the user, you use a form and JSON representation (even if, in retrospect, it is not the most efficient solution).

In this post, we'll take a different approach. We'll leverage some tooling to create a business language that extends the functionality of a Rails application.

Let's get started!

Background

A few years ago, my team implemented a feature that enabled users to input a set of complex conditions. Our system presented results in accordance with these conditions.

We took the conservative road and implemented a state-of-the-art form with multiple input widgets, drag and drop, and all the UI sugar. Then the form state was serialized into a JSON object and sent to the backend, where it was unpacked, conditions applied, and results sent back.

It worked, but not without multiple problems:

• The form was difficult to maintain, and bugs kept creeping in.
• It was also very complex; most users could only use 10% of its capabilities.
• Any change to JSON representation had to be implemented in two places: in the form frontend and the backend.

I will now discuss a possible alternative approach that can solve at least some of the problems outlined above and that we had not considered at the time.

A Problem to Solve in a Rails App

First, let me describe the problem we will solve in more detail. This is a very simplified version of the actual requirement I talked about above.

We'll build a backend for a promo mobile app, with a 'Coupons' section. I'm sure you are familiar with this concept from some real-world mobile applications as well.

At any given moment, you'll usually have a small number of coupons available — let's say, 10 to 30. This number is too large to fit on the screen, so to increase the conversion (usage) rate, it is important to personalize the order of the coupons. The ones most likely to pique a user's interest should go first, based on data about the available user.

So, when adding a new coupon to the system, the operator fills in the 'targeting info', i.e., the cohort this should interest. This might be based on science, intuition, or stereotypes. It can be very simplistic ("women aged 18–35") or quite complex ("women aged 50+ interested in health, or men interested in sports or the outdoors, or people with children aged 10+"). In fact, any combination of data assertions we have should be possible with any logical operators.

Using the aforementioned JSON representation, we could represent a complex condition with something like this:

{
  "operator": "or",
  "conditions": [
    {
      "operator": "and",
      "conditions": [
        { "property": "gender", "operator": "in", "value": ["woman"] },
        { "property": "age", "operator": "gt", "value": 18 },
        { "property": "age", "operator": "lt", "value": 35 }
      ]
    },
    {
      "operator": "and",
      "conditions": [
        { "property": "gender", "operator": "in", "value": ["man"] },
        {
          "property": "interests",
          "operator": "intersect",
          "value": ["sports", "outdoors"]
        }
      ]
    },
    {
      "property": "child_birth_year",
      "operator": "between",
      "value": [2004, 2012]
    }
  ]
}

Even though this is legible, it is hard to follow. And the form supporting it is tricky to understand as well (actually, we haven't even come up with a satisfactory solution for a form that mixes "and" and "or" logical operators).

A Solution: Design a New Business Language

An alternative solution to this problem is to get rid of the form and the JSON representation. Instead, we'll design a whole new, specific business language to apply here and then implement it in a Rails application. As a result, our condition will look like this:

(gender(woman) and age > 18 and age < 35) or (gender(man) and (interest(sports) or interest(outdoors))) or has_child_aged(10, 18)

As this is much terser, it is easier to understand and debug. It could be really hard for a regular user picked from the depths of the Internet to use. However, in cases like this, the actual users can be trained with access to fast customer support.

We could call our business language a domain-specific language (DSL) because this is more or less the term's original meaning. However, in recent years, especially in the Ruby community, the meaning of DSL has somewhat changed. When we talk about DSLs, we usually mean bending the Ruby language to look like something else while it is still actually Ruby. Think about RSpec:

describe "a subject" do
  let(:number) { 15 }
  it { expect(NumberDoubler.call(number)).to eq(30) }
end

Even though we introduce specific words defined as Ruby methods (describe, let, it), this is still Ruby.

It is important to understand that the business language example above is not Ruby. Users won't be able to access the filesystem, the network, or make infinite loops. All the language constructs only allow talking about users and their properties or combining expressions with logical operators.

It is a completely new language, which needs to have a grammar, parser, etc. This is what we are going to do next.

Implement a Business Language in Your Rails App with Parslet

To achieve our goals, we will use a gem called Parslet, for:

Constructing parsers in the PEG (Parsing Expression Grammar)
fashion.

Source: Parslet website

PEG is relatively simple (compared to other techniques) and quite fast, especially for small input.

This post is not going to be a step-by-step Parslet tutorial. The official Parslet documentation does a great job of going through the process. But let's briefly go over the building blocks of language interpreting with Parslet, which consists of three steps.

In step one, we define a parser:

class CouponLang::Parser < Parslet::Parser
  rule(:lparen)     { str('(') >> space? }
  rule(:rparen)     { str(')') >> space? }
  # [...]
end

tree = CouponLang::Parser.new.parse("age > 18 or age < 30")

The result of the parse method is an intermediary tree, which is a hash-like structure representing language tokens.

This tree is fed to a transform:

ConjunctionExpr = Struct.new(:left, :right) do
  def eval(user)
    left.eval(user) && right.eval(user)
  end
end

# [...]

class CouponLang::Transform < Parlet::Transform
  rule(:and => [subtree(:left), subtree(:right)]) { ConjunctionExpr.new(left, right) }
  # [...]
end

tree = CouponLang::Parser.new.parse("age > 18 or age < 30")
ast = CouponLang::Transform.new.apply(tree)

The result of the transform is an abstract syntax tree (AST). The final step is to evaluate the AST, passing a user as a context:

user = User.find(params[:user_id])
coupons = Coupon.active(Time.now)
proritized, others = coupons.partition do |coupon|
  tree = CouponLang::Parser.new.parse(coupon.priority_condition)
  ast = CouponLang::Transform.new.apply(tree)
  ast.eval(user)
end

The last code listing shows how this can be used in a wider context, relevant to our original requirements.

However, there is one important limitation to mention. As you can see, all active coupons are fetched upfront, and we apply the conditions on them one by one. It is safe, because, as stated before, only a handful of coupons are active at the time. However, this makes it impossible to answer the question: "What users would coupon X prioritize?"

If you really need to answer this kind of question, you have to fetch all the users and apply the conditions to them, user by user, in the application layer.

Of course, there are more efficient solutions.

For example, you can write another transform which, instead of evaluating the conditions, translates them into an SQL query which you can then execute.

A Complete Example

I'm not going to lie to you: getting the code parser for the language right might be a tedious task. I spent a few hours getting the example for this article working. For brevity, I show an example that only implements the age and has_children functions for the user params.

This is the parser:

module CouponLang
  class Parser < Parslet::Parser
    rule(:lparen) { str("(") >> space? }
    rule(:rparen) { str(")") >> space? }

    rule(:space) { match('\s').repeat(1) }
    rule(:space?) { space.maybe }
    rule(:sep) { space | any.absent? }

    rule(:or_) { str("or") >> space }
    rule(:and_) { str("and") >> space }

    rule(:gt) { str(">").as(:gt) >> space? }
    rule(:lt) { str("<").as(:lt) >> space? }
    rule(:comparison_op) { gt | lt }
    rule(:comparison) { int.as(:left) >> comparison_op >> int.as(:right) }

    rule(:integer) { match("[0-9]").repeat(1).as(:int) >> space? }
    rule(:string) { match["a-z"].repeat(1).as(:string) >> space? }

    rule(:age_fun) { str("age").as(:age_fun) >> space? }
    rule(:has_children_fun) { str("has_children").as(:has_children_fun) >> sep }

    rule(:int) { age_fun | integer }
    rule(:bool) { comparison | has_children_fun | bool_in_parens }
    rule(:bool_in_parens) { lparen >> expr >> rparen }

    rule(:expr) { infix_expression(bool, [or_, 1, :left], [and_, 1, :left]) { |left, op, right| {op.to_s.strip.to_sym => [left, right]} } }
    root(:expr)
  end
end

And this is the transform:

module CouponLang
  class Transform < Parslet::Transform
    UserAgeFun = Class.new do
      def eval(user) = user.age
    end

    HasChildrenFun = Class.new do
      def eval(user) = user.has_children?
    end

    IntLit = Struct.new(:int) do
      def eval(user) = int.to_i
    end

    InfixOp = Struct.new(:left, :op, :right) do
      def eval(user) = left.eval(user).public_send(op, right.eval(user))
    end

    LogicalOr = Struct.new(:left, :right) do
      def eval(user) = left.eval(user) || right.eval(user)
    end

    LogicalAnd = Struct.new(:left, :right) do
      def eval(user) = left.eval(user) && right.eval(user)
    end

    rule(:age_fun => simple(:_)) { UserAgeFun.new }
    rule(:has_children_fun => simple(:_)) { HasChildrenFun.new }
    rule(:or => [subtree(:left), subtree(:right)]) { LogicalOr.new(left, right) }
    rule(:and => [subtree(:left), subtree(:right)]) { LogicalAnd.new(left, right) }
    rule(:left => subtree(:left), :gt => simple(:_), :right => subtree(:right)) { InfixOp.new(left, :>, right) }
    rule(:left => subtree(:left), :lt => simple(:_), :right => subtree(:right)) { InfixOp.new(left, :<, right) }
    rule(:int => simple(:int)) { IntLit.new(int) }
  end
end

It's important to thoroughly test the language you have created. Otherwise, some unpleasant surprises will be waiting for you when you want to change it in the future. An example of a spec for the parser looks like this:

RSpec.describe CouponLang::Parser do
  it "parses expression with parentheses" do
    expect(described_class.new.parse_with_debug("has_children and (age > 12)")).to eq(and: [{has_children_fun: "has_children"}, {left: {age_fun: "age"}, gt: ">", right: {int: "12"}}])
  end
end

And for the transform:

def parse_and_eval(code, user)
  tree = CouponLang::Parser.new.parse(code)
  ast = CouponLang::Transform.new.apply(tree)
  ast.eval(user)
end

RSpec.describe CouponLang::Transform do
  it "evaluates conditions with parentheses" do
    user_with_children = User.new(birth_year: 1980, child_birth_years: [2008, 2012])
    user = User.new(birth_year: 1981)
    code = "age > 65 or (has_children and age > 40)"

    expect(parse_and_eval(code, user_with_children)).to eq(true)
    expect(parse_and_eval(code, user)).to eq(false)
  end
end

Of course, for a full Rails integration, you must also validate whether a user puts a valid CouponLang code in a new coupon form. Parslet returns nil when it cannot parse the text, so it is as simple as this:

class Coupon < ApplicationRecord
  validate :correct_code

  def correct_code
    CouponLang::Parser.new.parse(code).present?
  end
end

Extra Parsing Tips

We are ready to ship the CouponLang to our internal users, with great flexibility for defining conditions.

The road to arrive here has not exactly been without bumps, so here are a few additional tips:

Start Small

Writing a parser is difficult. Start with the simplest possible language, make a parser for it, write tests, commit, and add another feature.

For the CouponLang, I first started with a language that could only evaluate logical conditions with true and false values (like false and true). Then I added parentheses — (true or false) and true. Only after this was I ready to replace true/false literals with comparisons and functions.

Test Subparsers

I haven't shown it, but Parslet allows you to test only a subset of parsers. With our language above, you could use CouponLang::Parser.new.comparison.parse("11 > 12") to just check if the comparison part is fine. This is very useful for testing and debugging.

Check the Tree is Transform-friendly

Unless you are quite experienced in writing PEGs with Parslet, you may end up with a parser that works, but a tree that's not transform-friendly. As a result, you will have to change it. Be prepared for that.

Be Careful with Backward Compatibility

When changing the parser breaks backward compatibility, chances are that the coupons already saved in the database will stop working. You should consider running every coupon from the production database against the new parser to check whether they will work after the change. It's a one-time
operation, but important to save a lot of headaches.

When You Should Consider Using a Parser

You may think that my coupon example is quite unusual, even though it comes from a real-world problem. It's true that what I show here is not for everyday Rails development. You need a good reason to implement such a powerful (and not very user-friendly) tool as Parslet.

And the reason is simple — it is still easier than trying to do it any other way.

A few examples of when a parser might be worth considering include:

• For very fine access control to some resources - e.g., when you need to make a document accessible only for "all" managers, members of team alpha or beta that have worked here for at least 3 months, and Jason from HR".
• When modeling game-like conditions - Want to wield the Epic Battleaxe of Doom? Sure, you just need to be at least level 32, have at least 180 strength, and have finished the quest "Waiting for Ragnarok" or "The Abyss of Destiny".

Remember that you are not limited to languages that return a boolean as a result. You can, for example, create a language that evaluates numbers and thus gives multiple rankings for users. Or restaurants. Or dogs.

If you are brave enough, you could also build a language that behaves much more like a "real" programming language and executes something with variables and conditionals. For example, if you build a system that reacts to certain events (like a notification that a database is down):

if(notification.severity <= 2) {
  user = sample_from_groups('sre)
  if(user.has_phone_number and notification.severity == 1) {
    send_sms_to(user)
  } else {
    send_email_to(user)
  }

  send_slack_notification('alerts, notification.payload)
}

This, however, is much more complicated than the example I showed in this article. You might want to check out this attempt to parse Java source code with Parslet.

Wrapping Up

In this post, we saw how you can leverage tooling to build a programming language that extends your Rails application's functionality. Even if you don't need it in your current project, it's worth knowing that it is possible without too much effort.

We also explored when it's worthwhile to consider using this approach.

If this piqued your curiosity about creating languages, I hope you will have a lot of fun experimenting.

Happy parsing (and interpreting)!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!