DEV Community: alviomer

A/B Testing in Rails

alviomer — Wed, 07 Dec 2022 05:52:24 +0000

With customer retention and engagement as the goal business try different forms and techniques in order to attain maximum conversion and retention on any digital platform. Ranging from making the most complex decisions about an articulate user experience to minor things like changing the colour and positioning of text, images, buttons etc. This gave rise to the concept of split testing or commonly referred to as A/B Testing in the world of product development.
Recently I explored Rails ability to perform an A/B Test with in the framework. Surprisingly it turned out to be super easy and flexible to spin up an A/B Test experiment in rails using it's gem called Split.
This blog is all I did to enable rails application to perform an A/B Test on any scenario.

All right, first things first some of the key ingredients to make this recipe work are below:

Ruby on Rails
Redis as your cache engine
Split

Once you get all the ingredients right all we need is to start the preps to make this easy recipe.

This blog assumes that you already have ruby on rails as the platform up and running along with Redis as its cache store.

Below is the outline of this tutorial:

First we need to install the split gem.
Make some configurations to spin up our A/B Test.
Instantiate the test.
Finish the test.
Explore the results.

Step I: Installing Gem

Crux of empowering your rails application to perform A/B Test is to install this gem which will do wonders for you. Pretty straightforward nothing too complex in that I reckon.

gem install split

Hurray! We are done with enabling our rails to perform any sort of A/B Test.

Step II: Essential Configurations

Next up is doing all the required or any additional configuration that need to be done in order to achieve the desired outcome. I have divided configurations in two parts

Experiment Configurations

These set of configurations involve everything we need to get our split test up and running. From defining experiments to deciding the type of algorithm you wish to use. For this we first create our split.rb file in our rails initializers folder. I've used below as part of my configurations you can get a full list of configurations from Split homepage. I'll try to explain the ones I used for my work.

rails_root = Rails.root || "#{File.dirname(__FILE__)}/../.."
rails_env = Rails.env || 'development'

split_config = YAML.safe_load(ERB.new(File.read("#{rails_root}/config/split.yml")).result)
url = "redis://#{split_config[rails_env]}"

Split.configure do |config|
  config.redis = url  # this setting configures redis URL
  config.db_failover = true # important to define redis failover mechanism
  config.allow_multiple_experiments = true
  config.enabled = true

  config.persistence = :cookie # cookie persistence recomended for logged out users 
  config.persistence_cookie_length = 86400 # 1 day of cookie expiry time

  config.include_rails_helper = true
  config.store_override = true

  config.experiments = {
    :first_experiment => {
      algorithm: 'Split::Algorithms::Whiplash',
      alternatives: [
        { name: 'alternate_one', percent: 80 },
        { name: 'alternate_two', percent: 20 }
      ]
    }
  }
end

You might find this a bit overwhelming but I will explain everything I used in this.
First four lines of the file is just to fetch Redis URLs for different environments.

PS: I admit I could have made it much simpler but here I'm a lazy developer.

Each line wrapped in Split.configure do |config| means:

redis - This is the URL of the Redis server normally something like redist_host:redis_port.
db_failover - It is important to define what will happen in case of Redis failure, and to gracefully switch.
allow_multiple_experiments - This allows rails to run multiple experiments.
persistence - This property defines the persistence mechanism you want to use for your experiments. Generally for a logged out user cookies are used while for a logged in user, session adapters are used. However you can create your own customised adapters as well. More on this can be found on Split homepage.
persistence_cookie_length - Defines the length of cookie persistence, and its expiry the unit is seconds.
store_override - This is set to false if you don't want the statistics to get effected while you are forcefully testing an alternative.
experiments - This defines your experiments, here we've declared one experiment with the name first_experiment which has two alternates alternate_one & alternate_two, and is using Whiplash as the algorithm to select the instance of test while having a weights of 80 & 20 percent respectively.

The experiments in this configuration file can be defined in a seperate YML file as well, if there are multiple experiments to be configured.

Dashboard Configurations

Next we need to spin up an instance of a standalone dashboard that comes with this gem, where we can monitor all the tests that we have instantiated and how each of them are performing. In the dashboard we have the ability to stop an experiment where it will default back to just one of them. While you can even declare a winner and it will change everything on runtime. The split dashboard is mounted on your rails route so here is the configuration I used to mount it.

  match "/split" => Split::Dashboard, anchor: false, via: [:get, :post, :delete], constraints: -> (request) do
    request.env['warden'].authenticated?
    request.env['warden'].authenticate!
    request.env['warden'].user.admin?
  end

The above addition in routes.rb means that it will allow you to access split dashboard where you can monitor all the cool stuff happening on the go. However it is important to get this dashboard behind a security mechanism, so that it can not be accessed from outside. For this we have multiple options, we can be the best judge ourselves of how secure we want it to be some of the options are:

Using basic rack authentication mechanism (very basic and simple)
Using Devise based authentication mechanism
Using Warden based authentication mechanism In my configuration after defining split URL, I've given the access to authenticated admin accounts only. Now you can access the split dashboard using localhost:3000/split.

Step III: Instantiating a Test

Yay!! We have done the hard yard now time to instantiate our first test based on the configurations we did in the above sections. In order to do that all we need is to call the method ab_test this method takes the name of the experiment (defined in the configuration) as the first argument, and the variations can be passed as successive arguments.

ab_test("name_of_experiment", "alternate_1", "alternate_2")

You can get the A/B test running either in your views or your controller depending on the requirement:

Views

Let's say you want to change the text on your link, here's how you do it:

<a href="/url"><%= ab_test "link_experiment", "View", "Please View" %></a>

This can be modified to change anything on the go, like colour of your button, your partial views or your any of your html content.

Controllers

If you want to perform a test in your controller code we can use the same method to take different course of action within your controllers as well. Here is code snippet for that:

@ab_test = ab_test(:first_experiment, 'alternate_one', 'alternate_two')

The function will return one of the alternate and from there we can define two different course of actions that we intend to perform for each outcome, as shown in the below code.

@ab_test = ab_test(:first_experiment, 'alternate_one', 'alternate_two')
if @ab_test == 'alternate_one'
  # do something for alternate one
else
  # do something else for alternate two
end

Step IV: Finishing a Test

Yay!! Now that we have instantiated the test we need to eventually learn which alternate had a better chance of conversion, for this upon successful conversion of the alternate we need to end the test that we started. So that we can track the number of conversions for each alternate. In order to end the test for that session we can use below code snippet.

ab_finished(:first_experiment)

This function takes single argument which is the name of the experiment (defined in the configuration).

Explore Conversions

By now our split test is up and running and serving different variations to users based on the configurations we have chosen. We can give this a test by running our scenarios.

Question: Wait! What if I have to test a particular scenario, do I have to try my luck every time by refreshing the browser?
Answer: Absolutely Not! All we need to do is we need to pass the scenario in our URL to view a particular one. Something like this http://localhost:3000?ab_test[first_experiment]=alternate_two. This will serve you the second alternate we defined in our configuration.

The standalone dashboard comes quite handy in exploring the results and quickly perform some actions. Here is the view of the dashboard that we mount on rails routes.

Reducing Stress as Product Engineer & running smooth operations

alviomer — Sun, 14 Jun 2020 02:34:35 +0000

As a product engineer, and playing part in different capacities and lifecycle of a product myself I understand how the work we do enhance the level anxiety & stress at times, that takes over and effect our day to day lives. Be it the fear of new releases or deployments, be it an exception or user error coming up on production environment or be it the deadly production servers not responding or my personal pet peeve any issues regarding user payments, these fears and pressures are somehow part and parcel of being a product developer/engineer. When I say that these things are somehow part of every developer’s life, we can somehow find ways to mitigate them and make our lives less stressful than what it normally tends to be.
For this very reasons I have few observations that I found quite useful as I have been using them for quite some time now, and they certainly have made a positive impact on the business as well.

Test Driven Development

Where the concept of test driven development has been taking rounds in the arena for a while now, I still find developers not keen to adopt it (sometimes due to tight timelines for deliverables). I would say it is a life saver for sure, as it eliminates lot of edge cases which normally as humans we tend to miss out while developing. The phenomenon is all about developing strong test cases first, based on the business logic, then building the application code around it.

On-demand Cloud Infrastructure

From managing scale up/down polices to raising alerts under load conditions, this phenomenon gives you and your business a sigh of relief. The idea revolves around the concept of pay as you go and it has been a super saver for businesses with dynamic needs and ever changing ecosystem.

Observability & Monitoring

Gaining prime insights of any system is becoming crucial every passing day, it has never been as important to collect and analyze matrices and logs from a system than in today’s age. Thus giving birth to the concept of observability. Due to ever increasing and ever changing customer demands the operational visibility and the ability to monitor a system at any point in time has been of prime importance for IT and DevOps engineers.

Error monitoring tool

This is why tools that enable us to track all errors and exceptions of any system comes in handy for a smooth operation of any product. These tools enable engineers to remain one step ahead of any disastrous situation that may occur due to any unforeseen exception or error during the normal operation of a business.

Performance monitoring tool

Getting insights to the physical parameters of any system also comes under the umbrella of observability and this also provides an edge for the business operation team. Engineers must be constantly aware of infrastructure utilization of their product, in order to be a step ahead and take relevant actions.

CI/CD

There is nothing that makes me happier these days than a reliable and robust implementation of continuous integration & continuous delivery mechanism. Tools like Jenkins certainly gives you more power as a DevOps engineer, and has greatly reduced the amount of efforts used to deploy and test any new feature, without the worry of systems and normal operations coming to a halt. Rolling out new features with zero downtime is a dream come try for any business.

Adapting to changes & constant upgrades

I personally cannot stress more to this final point, as this is of utmost importance for the smooth execution of any operation. There has to be a will originating from the business owner to keep the systems up to date at any given point in time. New tools and technologies if applied at an early stage save developers from the later catastrophe of migrating them when the system you are using is completely obsolete.

Synopsis on Performance optimisation & Load Handling

alviomer — Thu, 11 Jun 2020 07:54:23 +0000

There is a common myth amongst businesses that performance testing and eventually scaling up systems according to the needs is all about enhancing the physical parameters of the infrastructure without being cognisant of many other factors. Trust me that’s not the case. There are many factors attached while applying an architectural scale up strategy to any system in today’s day and age. Merely scaling up physical infrastructure may give you a sense of security for a while, but in long run that is not sustainable enough unless we consider lot of other factors that are easily missed otherwise.

For past few years I have been closely performing tasks for performance enhancements on different projects with different scopes and varied complexity/scale. Here is the quick wrap up of some key considerations that I feel might be helpful while planning to scale up systems.

Anticipation of the expected load

Something that normally businesses tend to overlook is that apparently there are lot of limitations in terms of resources, cost, etc. due to which one cannot optimise a system to bare the load of whole universe coming at the same point in time. There are lot of factors that we need to consider before carrying out the activity and anticipation of approximate load is of prime importance. It doesn’t mean that the business has to give the accurate figure, but the tech team requires approximate load that is expected for the system usage.

This helps in scaling up the system considering the cost that will be incurred keeping in mind the ROI, which can be crucial for the business.

Tool to fire load organically on the system

After anticipation of an approximate load that is expected on the system one need to get hold of a tool to mimic organic load to the system. There are lot of tools available to carry out load testing on the system, my personal favourite being JMeter (of course that’s open source).

Gradually increasing test load on the system

I know jumping to conclusion is an impulsive human nature, but this is where we should hold our horses. We should start a gradual process, starting with an initial load and thereby increasing it while monitoring how the system responds under different load conditions. This is where the patience of the team is tested, because when we start increasing the load things will start to fall apart and the system will start behaving in an unexpected way, and this where next points will come handy.

Performance monitoring tool

One important tool that we require here is something that gives an insight about the current system resources and how the system is managing parameters like (just to name a few):
a. CPU utilisation
b. Memory utilisation
c. Number of concurrent requests being processed in the system
d. DB connections

Lot of COTS solutions are available for the purpose, that can be utilised to monitor, again my personal favourite being New Relic.

Error monitoring tool

Of course, no one wants to strain their eyes by sticking them on logs, for this reason it is of prime importance to invest in a tool that monitors exceptions and error conditions arising in system. Plus this is something that will come handy not just during load testing but also during normal operations of the business and allows tracking and regression for issues.

Optimisation Process

This is a two-step process normally this is where tech team tends to make a mistake and straight away jump to step 2. May be to accelerate the process or sometimes because of stringent timelines, but anyways this is where we need to calm ourselves and make a sage decision to invest some time and divide the optimisation process at two levels.

Code Level Optimisation & refactoring

Code level optimisation should be the first level that we should start with as this is where most of the benchmark is achieved. On this step the goal should be to make judgements about the most consumed parts of the system i.e. any piece of code or API that is being highly used/executed during the operations should be optimised and the output should be an efficient piece of code. It may require code refactoring or may be some alterations in the usage of external plugin if there are any.

Infrastructure Level Optimisations

After completely optimising and building a robust and efficient code base now is the time to jump on tweaking infrastructural level parameters. This requires tweaks in number of clusters, number of CPUs or the amount of Memory required. Further it also includes enhancements at DB connections threads per CPU or number of concurrent requests to be supported by the system.

On-demand Cloud Infrastructure (Optional)

Last thing which I personally recommend highly during these times where business is all about managing cost efficiently and effectively, is to use an on-demand cloud infrastructure. This will really help boost up an efficient infrastructure that is also cost effective at the same time.