DEV Community: Arjun Rajkumar

Stop Chasing Five Stars. Start Building Systems.

Arjun Rajkumar — Mon, 05 Jan 2026 05:49:39 +0000

There's a disease in small business. It's called "review anxiety."

Business owners refresh their Google Business Profile like teenagers checking Instagram likes. They lose sleep over a single three-star review. They beg customers for feedback in ways that make everyone uncomfortable. I've been thinking about this a lot lately while exploring tools like ReviewCow that try to solve this problem systematically.

This is backwards. And if you're a developer or someone who thinks in systems, you already know why.

The Real Problem Nobody Talks About

Here's what most business owners get wrong: they treat reviews as an outcome to chase, not a system to build.

They ask: "How do I get more five-star reviews?"

Wrong question.

The right question is: "How do I build a machine that naturally generates feedback, surfaces problems before they become public, and makes happy customers want to talk about us?"

That's a systems problem. And systems problems have systems solutions.

I've watched this play out hundreds of times. The businesses that win at reviews aren't the ones obsessing over star counts. They're the ones who built feedback into their operations the same way they built payroll or inventory management. It just runs.

Why Developers Should Care About This

If you're building software, you understand something most business owners don't: the difference between a manual process and an automated one.

Think about how we handle errors in code. We don't wait for users to email us about crashes. We instrument everything. We set up logging. We get alerts. We know about problems before customers do.

Reviews should work the same way.

The businesses crushing it with Google reviews have figured this out. They're not asking employees to remember to request reviews. They're not hoping customers will take initiative. They've built systems where feedback collection happens automatically, negative experiences get caught internally, and only genuinely happy customers end up on Google.

According to research from BrightLocal, 98% of consumers read online reviews for local businesses. But here's the stat that matters more: businesses that respond to reviews and actively manage their reputation see measurably higher conversion rates. The game isn't about vanity metrics. It's about building trust at scale.

The Employee Problem

Here's where it gets interesting.

Most review strategies fail because they depend on employees remembering to do something. "Hey, can you ask that happy customer to leave us a review?"

This is like asking developers to remember to write tests after they finish coding. Some will. Most won't. And the ones who do will be inconsistent.

The solution isn't motivation. It's automation and incentives.

Smart businesses are gamifying this. They're tracking which employees generate the most reviews. They're creating friendly competition. They're tying it to recognition, bonuses, or just good old-fashioned bragging rights.

Tools like ReviewCow exist specifically because this problem is so common. You can't expect a cafe or salon to build custom software for tracking employee-driven reviews. But you can give them a system that does it automatically.

The principle applies everywhere: remove human memory from the equation. Make the right behavior the easy behavior.

The Interception Pattern

Here's a pattern every developer will recognize: middleware.

In web frameworks, middleware sits between the request and your application. It can intercept, modify, redirect. It's one of the most powerful patterns in software.

The best customer feedback systems use a similar principle — but not in the way you might think.

You're not intercepting reviews. That's review gating, and Google will penalize you for it. Everyone sees the Google review link. Always. No exceptions.

But here's the clever bit: when someone indicates a low rating, you also ask for additional private feedback. "We're sorry to hear that. Tell us what went wrong so we can make it right." They still see the Google button. You're not hiding it. You're just opening a second channel.

Think of it as running two processes in parallel. The Google review option is always there. But unhappy customers now have a clear path to reach you directly if they want to. Many will choose that option because they'd rather get their problem solved than vent publicly.

You're not filtering anyone. You're not blocking the review link based on sentiment. You're just making sure both doors are open — and making the private door more visible when it matters most.

What This Looks Like In Practice

Let me paint a picture.

A dental practice uses QR codes in their reception area. Scan it, you get a simple five-star rating widget. Click any rating, and you see the Google review button.

But if you clicked 1, 2, or 3 stars? You also see a feedback form. "We're sorry to hear that. Tell us what went wrong." Your name, your comments, straight to the practice manager.

The Google button is still right there. You can click it. Nobody's hiding it. But now there's an obvious alternative: talk to us directly so we can actually fix your problem.

This isn't review gating. Everyone sees the Google link regardless of their rating. You're just making the private feedback option more prominent for people who are unhappy. Most of them would rather get their issue resolved than leave a angry review that goes nowhere.

The result: happy customers breeze through to Google. Unhappy customers have a clear path to resolution. And you get an early warning system for problems before they spiral.

This is what I mean by building systems. You set it up once, and it runs. Like a cron job for customer happiness — one that plays by the rules.

If you're in the business of building tools for small businesses, there's a massive opportunity here. Most local businesses are still doing this manually, if they're doing it at all. Platforms like ReviewCow are starting to fill this gap, but the market is huge and underserved.

The Uncomfortable Truth

Most businesses don't have a review problem. They have an operations problem that shows up in reviews.

If you're consistently getting negative feedback about slow service, the answer isn't better review management. It's faster service.

Reviews are a lagging indicator. By the time they show up, the damage is done. The real work happens upstream.

So yes, build systems for collecting reviews. Automate the ask. Gamify employee participation. Intercept negative feedback before it goes public.

But don't forget the thing that actually matters: being good at what you do.

No amount of clever systems will save a bad product. Systems just amplify what's already there. If you're great, they'll help more people find out. If you're not, they'll just surface problems faster.

Which, honestly, is still valuable. Problems you know about are problems you can fix.

Build the system. Then deserve the reviews.

Understand Ruby Blocks by looking at examples from popular gems

Arjun Rajkumar — Fri, 28 Oct 2022 09:43:35 +0000

Recent articles from Progress Updates:

Ruby Blocks is something all Ruby developers use everyday. For example, while using the each, times, or upto methods or the countless other Enumerable module methods, we normally pass a block along with the method. For example:

1.upto(5) {|i| puts(i)}

The code included in the brackets {} is passed to the upto method. This isn't passed as an argument, but as a block, and it is yielded by the upto method.

There are many different ways to use blocks, and here are some common patterns for using blocks:

1. Execute around

This pattern is mainly used when you have some common code you need to run before or after something else. For example, suppose you have a benchmark method like below:

def with_benchmark
  start_time = Time.now
  puts "Starting benchmark at #{start_time}"

  yeild

  stop_time = Time.now
  puts "The method took #{stop_time - start_time} to complete"
end

Now you can find out how long any method takes to complete by doing something like this:

with_benchmark do
  # Call the signup method  
end

with_benchmark do
  # Call any other method to find out how long it takes to run 
end

You could even improve the benchmark method to take an argument, and compare that with the result of the yield. For example:

def with_benchmark(expected_time)
  start_time = Time.now
  puts "Starting benchmark at #{start_time}"

  result = yeild

  stop_time = Time.now
  puts "The method took #{stop_time - start_time} to complete"

  if result > expected_time
    puts "Failed! Took too long to complete"
  else
    puts "Passed! That was super fast!"
  end
end

with_benchmark(5) do
  # Do anything you want
  sleep 10
end

with_benchmark(5) do
  # Call some method
  sleep 2
end

Examples of 'Execute around' from gems:

The Ruby standard library has a benchmark module that includes methods similar to what we have written above:

def realtime
  r0 = Time.now
  yield
  Time.now - r0
end

It is not necessary that we have to run something before and after the yield. Sometimes, we can run only the before part, or only the after part. For example, you may need to change the state of something before yielding. The Shopify App gem maintained by Shopify, in the activate_shopify_session method, activates a Shopify session before yielding the block.

module ShopifyApp
  module LoginProtection
    ...
    def activate_shopify_session
      ...
      begin
        ShopifyAPI::Context.activate_session(current_shopify_session)
        yield
      ensure
        ShopifyAPI::Context.deactivate_session
      end
    end

Like in the example above, it is also always recommended to use an ensure clause to change the state back incase the yielded block raises an error. As the code in the ensure clause is always executed, whether or not an exception is raised, it's guaranteed to return the state to how things were before you called yield.

2. Initializing an object with default values

Another common pattern of using blocks is to initialize an object with default values.

In Rails, you can create a new object in many different ways:

# By passing the attribues as a hash
Shop.new(name: "Starbucks")

# By setting the attributes after initializing or creating the object
shop = Shop.new
shop.name = "Starbucks"

# Or by using blocks!
shop = Shop.new do |s|
  s.name = "Starbucks"
end

While creating a new Ruby gem

Gem::Specification.new do |s|
  s.name        = 'example'
  s.version     = '0.1.0'
  s.licenses    = ['MIT']
  s.summary     = "This is an example!"
  s.description = "Much longer explanation of the example!"
  s.authors     = ["Ruby Coder"]
  s.email       = 'rubycoder@example.com'
  s.files       = ["lib/example.rb"]
  s.homepage    = 'https://rubygems.org/gems/example'
  s.metadata    = { "source_code_uri" => "https://github.com/example/example" }
end

This pattern is useful if you want to make the code more readable, and explicitly state what are the attributes required while setting up a new object.

To initialize an object with a block, you can do it by yielding the object on initialization. For example, if we had a simple Shop class, we could do it this way:

class Shop
  attr_accessor :name

  def initialize
    @name = "Starbucks"
    yield(self) if block_given?
  end

  def to_s
    "Name: #{@name}"
  end
end

shop = Shop.new
puts shop
# (would output "Name: Starbucks")

shop = Shop.new do |s|
 s.name = "Coffee Day"
end
puts shop
# (would output "Name: Coffee Day")

3. Saving blocks to execute later

This is something I came across while reading Eloquent Ruby. This pattern is useful when you want to use blocks to be called only when required - i.e. you can hang on to a block until you need to use it.

Until now we have called blocks implicitly using the yield to fire off the block. However, blocks can also be captured as a parameter, and can then be run by calling the call method on it. Calling blocks explicitly has it's own advantages and is clearer to someone reading the code.

This is an example from the Pay gem.

module Pay
  module Webhooks
    class Delegator
      ...

      # Configure DSL
      def configure(&block)
        raise ArgumentError, "must provide a block" unless block
        block.arity.zero? ? instance_eval(&block) : yield(self)
      end
      ...
    end
  end
 end

Passing blocks explicitly can also be really useful if you have an expensive method, and you only want to run it when really needed. Eloquent Ruby, shows a useful application for this. Suppose you have a Document class, and most times you only need to fetch the title and the author. Occasionally, you may need to read the actual content of the document from different sources that are chosen by the user.

class Document
  attr_reader :title, :author

  def initialize(title, author, &block)
    @title = title
    @author - author
    @initializer_block = block
  end

  def content
    if @initializer_block
      @content = @initializer_block.call
      @initializer_block = nil
    end
    @content
  end
end

Implementing the document class this way means we can can get the contents from a file, or via HTTP, or via FTP.

file_doc = Document.new('file', 'Eloquent Ruby') do
  File.read('some_text.txt')
end

google_doc = Document.new('http', 'Eloquent Ruby') do
  Net::HTTP.get_response('www.google.com', '/index.html').body
end

The above examples does look similar to the initializing examples below - but the main difference is that the Document class waits until the content method to fire off the block. If the content method is never called, the block will also never get called.

We can also see more examples of this pattern where the block is saved for later in before_action in Rails.

class ApplicationController < ActionController::Base
  before_action do |controller|
    # do something before each action
  end
end

And in Active Record's life cycle hooks.

class User < ActiveRecord::Base
  before_create do |user|
    puts "about to create #{user.name}"
  end
end

Conclusion
Using blocks with yield may take some time to get used to, but it gives us a different style of programming, that can improve the design of our code.

If you are looking for a senior Rails developer to join your team, do send me an email to arjun.rajkumar@hey.com. I took 6 months off as I became a new dad, but am currently open to new Ruby/Rails related work.

This post originally appeared on Weightless.

Keeping my Ruby Class Methods small and focussed

Arjun Rajkumar — Fri, 30 Sep 2022 09:32:28 +0000

Recent articles from Progress Updates:

One of the benefits of programming in ruby is that it is easy to write short methods. This is helpful as it makes it easier to test different methods in isolation. Also, most importantly it's easier to read the class when the class is constructed of small methods. Often, when I get back to a program after a while, I may not have enough context, and writing short, focussed methods helps with this.

For example, in one of my Shopify apps, I let users add multiple filters to a workflow.

For example, when an order is created, I need to check the order with workflows conditions, and then add tags accordingly, if the check passes. For each of the conditions, I also need to check if the user added AND (all conditions must match), or OR (any condition must match), and then make a call to the Shopify API to get the data.

I like using service objects for handling the business logic for the app. The service object below is the one that checks if the workflow conditions passes or not.

class CheckWorkflowConditions
  attr_accessor :shop

  def initialize(shop:, workflow:, order_id:)
    @shop = shop
    @workflow = workflow
    @order_id = order_id

    @filter_check_results = []
  end

  def passes?
    return false if no_filters?

    return atleast_one_filter_passes? if any_conditions_filter?
    return all_filters_passes? if all_conditions_filter?
  end

  private

  def no_filters?
    @workflow.filters.empty?
  end

  def any_conditions_filter?
    !all_conditions_filter?
  end

  def all_conditions_filter?
    @workflow.all_conditions
  end

  def atleast_one_filter_passes?
    @workflow.filters.each do |filter| 
      perform_filter_check(filter)
      break if includes_passed_filter?
    end
    includes_passed_filter?
  end

  def all_filters_passes?
    @workflow.filters.each do |filter| 
      perform_filter_check(filter)
      break if includes_failed_filter?
    end
    !includes_failed_filter?
  end

  def perform_filter_check(filter)
    @filter_check_results << CheckOrderWithFilter.new(shop: shop, order_id: @order_id, filter: filter).get_result
  end

  def includes_passed_filter?
    @filter_check_results.include?(true)
  end

  def includes_failed_filter?
    @filter_check_results.include?(false)
  end
end

The CheckWorkflowConditions service object has followed the composed method technique from Eloquent Ruby. That is:

Each method does one single thing. The no_filters? method only checks if filters are present? Similarly all the other methods in this class focusses only on doing one thing each.
Each method operates at single conceptual level. The high level logic is not mixed with the details in any method.
Each method has a name that specifies what it does. The method name kind of tells us what the method is trying to do.

The above service object also does not follow the advice that each method should only have one way out, so that all the logic results in a return at the bottom of the method. Although the passes? method has multiple returns, because we have followed the composite technique, it is easy to read and understand.

Also, the logic related to making the Shopify API calls has been delegated to a different service object: CheckOrderWithFilter.new(shop: shop, order_id: @order_id, filter: filter).get_result. This makes the whole class more focussed on just checking if the workflow conditions pass or fail, making it easier to test.

Writing (and reading) ruby code becomes more enjoyable when the class is made of methods that are short, focussed.

This post originally appeared on Weightless.

Learnings from 9 months at a startup.

Arjun Rajkumar — Sat, 14 Nov 2020 01:40:43 +0000

Hi! I'm Arjun. Currently building ReviewCow

Your team mates shouldn’t have to read the PRD to get context about code they are reviewing. Name methods, variables, files in a way that gives more clarity to what it’s doing.
Focus on reducing request times. This can be done by moving non critical tasks to background jobs.
Related to this: Show something is being done to the customer ASAP while you do the main work in the background.
It’s good to challenge product teams requirements in the ideation phase as product folks don’t always think about (or may not be aware of) performance while deciding on upcoming features. Developers should call out things that could lead to poor performance.
While coding, avoid loops/iterating thru all to do calculations. Related: careful loading all records from DB. Example: to check if atleast 10 records exists, use User.first(10).size instead of User.count
If you have to update a lot of users manually, try doing it in real time instead of a rake task. Trigger the update when the user completes an action.
Use a linter/code formatter that everyone in the team follows.
Write tests for the code you write. Don’t have to test external gems behaviour.
Measure test coverage. Aim for > 80%
Use error monitoring alerts. Make it loud. Better to catch and fix errors initially before it starts affecting more users.
Use environment variables / public configurations. This way you don’t have to commit code to change things which could be changed via env variables.
Get warnings, performance error notifications while you code.
Also use external services for improving performance. Then identify slow requests and fix.
Don’t ignore staging environment warnings.
Get other people to review your code. It’s more efficient to get/give fed feedback in one go. Review/read other people PRs as you’ll pick up things you didn’t know before. While reviewing it helps thinking how this can go wrong.
Before writing any code, go through the requirement and break it into smaller tasks. Make a checklist and identify blockers if any. Clear the blockers before you start coding.
Avoid working on more than 1 task at a time. Less chances of coginitive overload.
People shouldn’t need to ask you what you’re doing. They should look at the todos, see the checklists and know the status of a feature.
Before coding, agree on contract between front end and back end developers.
Before coding, understand what the inputs and outputs will be. Example, if using an external API, test the API response with a curl request and share with product team or test with external services to see if this is what they expected.
Before coding, ask yourself how this will affect backward compatibility.
Write tests for backward compatibility.
Better to not code a feature rather than to code, and then realise it’s not working as expected.
Over communicate, especially with remote work.
Give uninterrupted time to developers to code.
Cultivate a culture of asynchronous communication. Most of the time you don’t need to reply to other people immediately. * Expectation of immediate replies, constant notifications decrease productivity.
But it’s important to have some time overlap among team members. I'd settle for ~ 2 hours of overlap a day.
Avoid meetings when possible. Needed if you are stuck with blockers.
Don’t work more than 8 hours most days. It’s hard to be consistent if you’re putting in 10+ hours or working till 3AM.
Avoid pushing to production during non work hours as it requires more people to be up.
Use emojis in slack. It build rapport.

GraphQL + SQL Magic for faster downloads on Shopify

Arjun Rajkumar — Fri, 03 Jul 2020 02:18:21 +0000

Hi! I'm Arjun. Currently building ReviewCow

Shopify's GraphQL Admin API lets you download large volumes of data asynchronously in bulk.

For example, here is a bulk query for getting a list of all the products with its images and variants.

def structure_the_query
    context.download_products_query = context.client.parse <<-'GRAPHQL'
      mutation {
        bulkOperationRunQuery(
         query: """
          {
            products {
              edges {
                node {
                  id
                  title
                  images {
                    edges {
                      node {
                        id
                        originalSrc
                      }
                    }
                  }
                  variants {
                    edges {
                      node {
                        id
                        title
                        position
                        price
                        image {
                          id
                        }
                      }
                    }
                  }
                }
              }
            }
          }
          """
        ) {
          bulkOperation {
            id
            status
          }
          userErrors {
            field
            message
          }
        }
      }
    GRAPHQL
  end

And, when the bulk query operation is complete, you get back a JSON file.. This is a sample of how it looks like

{"id":"gid:\/\/shopify\/Product\/5311829246111","title":"Guitars"}
{"id":"gid:\/\/shopify\/ProductImage\/17459197116575","originalSrc":"https:\/\/cdn.shopify.com\/s\/files\/1\/0414\/2294\/6463\/products\/shopping.png?v=1592737453","__parentId":"gid:\/\/shopify\/Product\/5311829246111"}
{"id":"gid:\/\/shopify\/ProductVariant\/34792999878815","title":"Default Title","position":1,"price":"0.00","image":null,"__parentId":"gid:\/\/shopify\/Product\/5311829246111"}

This is my flow on how I used GraphQL with Active Record Import to improve performance.

1. Retrieve all the products from the shop

def call
  activate_graphql_client
  structure_the_query
  make_the_query
  poll_status_of_bulk_query
  retrieve_products
end

The five methods above in the interactor focusses on making the connection with Shopify and retrieving the products. The retrieve_products method does the job of checking if the bulk query is completed, and saving the products to the database.

def retrieve_products
    if context.result_poll_status.data.current_bulk_operation.status == 'COMPLETED'
      context.url = context.result_poll_status.data.current_bulk_operation.url

      SaveProducts.call!(shop_id: context.shop.id, url: context.url)
    else
      sleep 2
      poll_status_of_bulk_query
      retrieve_products
    end
  end

2. Save the products in your database

The retrieve_products method calls this interactor: SaveProducts.call!(shop_id: context.shop.id, url: context.url) whose main purpose is to download the products from the file which Shopify shared with us, and to save the products in the database.

class SaveProducts
  include Interactor

  def call
    download_products_from_file
  end

  private

  def download_products_from_file
    context.products = []
    line = nil

    URI.open(context.url) do |f|
      f.each do |new_line|
        parse_line(line, new_line) if line
        line = new_line
      end
      parse_line(line)
    end

    Product.import context.products, recursive: true
  end

  def parse_line(line, next_line=nil)
    json = JSON.parse(line)

    if belongs_to_product(json['id'])
      create_new_product(line, json)

      move_product_into_products(next_line)
    elsif belongs_to_image(json['id'])
      create_new_image(line, json)

      move_product_into_products(next_line)
    elsif belongs_to_variant(json['id'])
      create_new_variant(line, json)

      move_product_into_products(next_line)
    end
  end

  def belongs_to_product(id)
    !belongs_to_image(id) && !belongs_to_variant(id)
  end

  def belongs_to_image(id)
    id.include? 'ProductImage'
  end

  def belongs_to_variant(id)
    id.include? 'ProductVariant'
  end

  def create_new_product(line, json)
    context.product = Product.new(shopify_product_id: json['id'].delete('^0-9'), title: json['title'], shop_id: context.shop_id)
  end

  def move_product_into_products(next_line=nil)
    if next_line
      json_next_line = JSON.parse(next_line)
      context.products << context.product if belongs_to_product(json_next_line['id'])
    else
      context.products << context.product
    end
  end

  def create_new_image(line, json)
    context.product.images.build(shopify_image_id: json['id'].delete('^0-9'), src: json['originalSrc'], shop_id: context.shop_id)
  end

  def create_new_variant(line, json)
    context.product.variants.build(shopify_variant_id: json['id'].delete('^0-9'), title: json['title'], position: json['position'], price: json['price'], shopify_image_id: json.dig('image', 'id')&.delete('^0-9'), shop_id: context.shop_id)
  end
end

This line Product.import context.products, recursive: true also helps in improving the performance of the downloads. This comes from the ActiveRecord Import gem.

If you had to download 1000 products, and were hitting the DB each time, you would end up making 1000 SQL calls. Instead Product.import lets you batch all the products together, so that you can save this with just 1 SQL call.

Move on from manually paginating results and managing a client-side throttle, to performing bulk operations with the Shopify's GraphQL Admin API. And combine this with some SQL magic to get some real performance boosts!

Building a Shopify App with Rails, React and GraphQL

Arjun Rajkumar — Tue, 16 Jun 2020 04:54:01 +0000

Shopify has some great tutorials on how to build apps using Node.js + React and Ruby and Sinatra - but the tutorials they have with Rails doesn't explain how to integrate it with React or GraphQL. And as Shopify is investing a lot in them, I decided to write this blog to help future developers who are looking to build an app using Rails, React and GraphQL.

I am going to walk you through my workflow on building a Shopify app with Rails and React, and using GraphQL to communicate between the two. We'll also use GraphQL to talk to the Shopify APIs. This post assumes that you already have setup Rails and React with Webpacker, and that you are familiar with GraphQL. If you are just starting out and need help setting up Rails, React or GraphQL, here are a few good resources.

High-level requirements

By the end of this tutorial, we are going to successfully import products from the Shopify Admin API and display it on our app. This list is a high-level breakdown of how we are going to approach this:

Connecting to Shopify
Retrieving product information from Shopify
Storing the products in your database
Displaying the products

Connecting to Shopify

I use two gems (both created by Shopify) to access the admin section programmatically. They provide the necessary controllers and all the required code for authenticating via OAuth. Do follow the steps mentioned in these gems to create an app, request access, and to get an access token.

You should also create the necessary models - Shop, Product and Image - to your app.

class Shop < ApplicationRecord
  include ShopifyApp::SessionStorage

  has_many :products, dependent: :destroy

  def api_version
    ShopifyApp.configuration.api_version
  end
end

class Product < ApplicationRecord
  belongs_to :shop
  has_many :images, dependent: :destroy
end

class Image < ApplicationRecord
  belongs_to :product
end

Retrieving product information from Shopify

The first thing to do when a new customer downloads the app is to retrieve all their products from the store. For this, we can use an after_create Active Record callback to automatically start the download.

class Shop < ApplicationRecord
  ...

  after_create :download_products

  def download_products
    Shopify::DownloadProductsWorker.perform_async(id)
  end

  ...
end

I do this via a background worker via Sidekiq. Most of the stores will have 100 - 1000s of products and you don't want to keep the user waiting while your app is downloading the products.

module Shopify
  class DownloadProductsWorker
    include Sidekiq::Worker

    def perform(shop_id)
      DownloadProductsFromShopify.call!(shop_id: shop_id)
    end
  end
end

The above worker delegates this process to an interactor. Interactors serve as a one-stop place to store all the business logic for the app. Another bonus is that it handles background failures and retries the worker easily. By default, Sidekiq only retries for StandardErrors. By moving all the logic to an interactor, and using .call! it throws an exception of type Interactor::Failure, which in-turn makes the Sidekiq worker to also fail, and re-try the job again for any error.

class DownloadProductsFromShopify
  include Interactor::Organizer

  organize ActivateShopifySession, DownloadProducts, DeactivateShopifySession
end

While downloading the products from Shopify, we have to first activate the session, download the products and then deactivate the Shopify session.

I've put this into an organiser which does these three steps one after the other. By separating these three requirements into their own classes, we can re-use them in other places.

Below are the two interactors for activating and deactivating the Shopify session.

class ActivateShopifySession
  include Interactor

  def call
    ActiveRecord::Base.transaction do
      find_shop
      create_session_object
      activate_session
    end
  end

  private

  def find_shop
    context.shop = Shop.find(context.shop_id)
  end

  def create_session_object
    shop = context.shop
    domain = shop.shopify_domain
    token = shop.shopify_token
    api_version = Rails.application.credentials.api_version

    context.shopify_session = ShopifyAPI::Session.new(domain: domain, token: token, api_version: api_version)
  end

  def activate_session
    ShopifyAPI::Base.activate_session(context.shopify_session)
  end
end


class DeactivateShopifySession
  include Interactor

  def call
    ShopifyAPI::Base.clear_session
  end
end

Downloading products from Shopify

The DownloadProducts interactor is responsible for downloading all the products from the Shopify store.

class DownloadProducts
  include Interactor

  def call
    ActiveRecord::Base.transaction do
      activate_graphql_client
      structure_the_query
      make_the_query
      poll_status_of_bulk_query
      retrieve_products
    end
  end
end

It connects to Shopify's GraphQL client, structures the query and gets the results from Shopify. With Shopify's GraphQL Admin API, we can use bulk operations to asynchronously fetch data in bulk.

class DownloadProducts
  ...  
  private

  def activate_graphql_client
    context.client = ShopifyAPI::GraphQL.client
  end

  def structure_the_query
    context.download_products_query = context.client.parse <<-'GRAPHQL'
      mutation {
        bulkOperationRunQuery(
         query: """
          {
            products {
              edges {
                node {
                  id
                  title
                  images {
                    edges {
                      node {
                        id
                        originalSrc
                      }
                    }
                  }
                }
              }
            }
          }
          """
        ) {
          bulkOperation {
            id
            status
          }
          userErrors {
            field
            message
          }
        }
      }
    GRAPHQL
  end

  def make_the_query
    context.result = context.client.query(context.download_products_query)
  end

  def poll_status_of_bulk_query
    context.poll_status_query = context.client.parse <<-'GRAPHQL'
      query {
        currentBulkOperation {
          id
          status
          errorCode
          createdAt
          completedAt
          objectCount
          fileSize
          url
          partialDataUrl
        }
      }
    GRAPHQL

    context.result_poll_status = context.client.query(context.poll_status_query)
  end

...
end

When the operation is complete, the results are delivered in the form of a JSONL file that Shopify makes available at a URL. We can use this URL to download all the products and images, and store them in our database.

require 'open-uri'

class DownloadProducts
  ...
  def download_products
    images = []
    products = []

    URI.open(context.url) do |f|
      f.each do |line|
        json = JSON.parse(line)

        if json.key?('originalSrc') 
          image_id = json['id'].delete('^0-9')
          image_product_id = json['__parentId'].delete('^0-9')
          image_url = json['originalSrc']

          image = Image.new(shopify_image_id: image_id,                  
                            shopify_image_product_id: image_product_id,
                            url: image_url,
                            shop_id: context.shop.id)
          images << image
        else
          prodcut_id = json['id'].delete('^0-9')
          prodcut_title = json['title']

          product = Product.new(title: prodcut_title,
                               shopify_product_id: prodcut_id,
                               shop_id: context.shop.id)
          products << product
        end
      end
    end

    Image.import images, recursive: true, on_duplicate_key_ignore: true
    Product.import products, recursive: true, on_duplicate_key_ignore: true
  end
end

Using GraphQl with the activerecord-import gem, improves the performance of the app. We can download 1000s of products and store them in the database, with just 2 SQL calls - one for bulk storing all the products, and one for storing the images.

GraphQL

Before we discuss the logic for downloading all the products, we need to talk about GraphQL. GraphQL is a query language for interacting with an API. Few advantage of GraphQL over REST APIs are

GraphQL only provides the data you ask for, reducing bandwidth and overhead, and usually improves the speed of your app.
Unlike REST APIs, which uses multiple endpoints to return large sets of data, GraphQL uses a single endpoint.
When downloading 1000s of products it is faster to download them via GraphQL's bulk queries.

Setting up GraphQL types and queries

I've used the following gems for working with GraphQL.

# GraphQL
gem 'graphql'
gem 'graphql-batch'
gem 'graphql-client'
gem 'graphql-guard'
gem 'apollo_upload_server', '2.0.1'

As we want to download products and images from a shop, we need to define GraphQL types for all them individually.

module Types
  class ShopType < Types::BaseObject
    field :id, ID, null: false
    field :shopify_domain, String, null: true
    field :shopify_token, String, null: true
    field :products, [Types::ProductType], null: true

    def products
      AssociationLoader.for(Shop, :products).load(object)
    end
  end
end

The AssociationLoader comes from graphql-batch, another gem built by Shopify, which is useful for handling N+1 errors on GraphQL.

Similarly, we also need to define the Product and Image Graphql Types.

module Types
  class ProductType < Types::BaseObject
    field :id, ID, null: true
    field :title, String, null: true
    field :shop, Types::ShopType, null: true 
    ...
    field :images, [Types::ImageType], null: true
  end
end

module Types
  class ImageType < Types::BaseObject
    field :id, ID, null: true
    field :url, String, null: true
    ...
    field :product, Types::ProductType, null: true 
  end
end

This allows us to create a ProductsResolver which can be used to query all the products from a shop.

module Resolvers
  class ProductsResolver < Resolvers::BaseResolver
    type [Types::ProductType], null: false

    def resolve
      context[:current_shop].products.includes(:images)
    end
  end
end

context[:current_shop] is being set in the GraphqlController.

class GraphqlController < AuthenticatedController
  before_action :set_current_shop
  before_action :set_context
  before_action :set_operations

  def execute
    if @operations.is_a? Array
      queries = @operations.map(&method(:build_query))
      result = ImagedropSchema.multiplex(queries)
    else
      result = ImagedropSchema.execute(nil, build_query(@operations))
    end
    render json: result
  end

  private

  def set_current_shop
    return if current_shopify_domain.blank?

    @current_shop ||= Shop.find_with_shopify_domain(current_shopify_domain)
  end

  def set_context
    @context = {
      current_shop: @current_shop,
      current_request: request
    }
  end

  ...
end

Display Products

Shopify Polaris is a style guide that offers a range of resources and building elements like patterns, components that can be imported into your app. The advantage of using Polaris is that you don't have to spend anytime building the UI, getting the colour etc correct - Shopify has already done all the hard work, and we don't need to worry about these details. The recommended way to use Polaris is via React.

I have build a React component that displays all the products with images, and provides search and sort functionalities. We are using useQuery to make the query via GraphQL to get list of products.

import React, { Component, useState, useEffect } from "react";
...
const PRODUCTS_QUERY = gql`
  query {
    products {
      id
      title
      images {
        id
        url
      }
    }
  }
`;

const Shop = () => {
  const { data } = useQuery(PRODUCTS_QUERY);
  const [products, setProducts] = useState([]);

  const [currentPage, setCurrentPage] = useState(1);
  const [searchQuery, setSearchQuery] = useState("");
  const [selectedCollection, setSelectedCollection] = useState(null);
  const [pageSize, setPageSize] = useState(10);
  const [sortColumn, setSortColumn] = useState({
    path: "title",
    order: "asc",
  });

  const handleDelete = (product, image) => {
    const products = [...products];
    const index = products.indexOf(product);
    products[index] = { ...product };

    const images = products[index].images.filter((i) => i.id != image.id);
    products[index].images = images;

    setProducts(products);
  };

  const handlePageChange = (page) => {
    setCurrentPage(page);
  };

  const handleCollectionSelect = (collection) => {
    setSelectedCollection(collection);
    setSearchQuery("");
    setCurrentPage(1);
  };

  const handleSearch = (query) => {
    setSelectedCollection(null);
    setSearchQuery(query);
    setCurrentPage(1);
  };

  const handleSort = (sortColumn) => {
    setSortColumn(sortColumn);
  };

  const getPageData = () => {
    let filtered = products;
    if (data) filtered = data['products'];

    if (searchQuery)
      filtered = filtered.filter((p) =>
        p.title.toLowerCase().startsWith(searchQuery.toLowerCase())
      );
    else if (selectedCollection && selectedCollection.id)
      filtered = filtered.filter(
        (p) => p.collection_id === selectedCollection.id
      );

    const sorted = _.orderBy(filtered, [sortColumn.path], [sortColumn.order]);

    const paginatedProducts = paginate(sorted, currentPage, pageSize);

    return { totalCount: filtered.length, pageData: paginatedProducts };
  };

  const { totalCount, pageData } = getPageData();


  return (
    <React.Fragment>
      <Navbar />
      <Layout>
        <Layout.Section secondary>
          <Sticky>
            <Game />
            <Dropzone />
          </Sticky>
        </Layout.Section>
        <Layout.Section>
          <div className="row">
            <div className="col-10">
              <SearchBox value={searchQuery} onChange={handleSearch} />
              <ProductsTable
                products={pageData}
                sortColumn={sortColumn}
                onDelete={handleDelete}
                onSort={handleSort}
              />
              <Paginate
                itemsCount={totalCount}
                pageSize={pageSize}
                currentPage={currentPage}
                onPageChange={handlePageChange}
              />
            </div>
            <div className="col-2">
              <ToastContainer />
              <ListGroup
                items={collections}
                selectedItem={selectedCollection}
                onItemSelect={handleCollectionSelect}
              />
            </div>
          </div>
        </Layout.Section>
      </Layout>
    </React.Fragment>
  );
};

export default Shop;

The Layout and Sticky components have been imported from Shopify Polaris.

Next steps

We have successfully imported products from the Shopify Admin API and displayed them on our app.

We used GraphQL to talk to Shopify's APIs and also to communicate between the Rails and React components in our app. In the next blog, we will explore adding a drag-and-drop functionality to the app, and also adding Shopify's billing API to collect payments.

Fast(er) tests with Stunt Doubles and VCR Recordings.

Arjun Rajkumar — Wed, 19 Feb 2020 08:10:19 +0000

Hi! I'm Arjun. Currently building ReviewCow

Making Tests Specific

One benefit of writing specific tests is that it requires your code to be well organised. For example, earlier on in the application, the code which talks talk to the Shopify APIs were scattered all over the application. But, because I wanted each of my tests to be specific and do only one thing, I had to move this part of the code which is used a lot in communicating with the APIs into its own section - a ShopifyWrapper module - whose only purpose is to interface with Shopify. By clearly having these application boundaries in my code, it helped with writing better tests. And in my application, I only need to interact with the Shopify wrapper - and the wrapper takes care of communicating with the Shopify APIs.

For example, in the background job below, I only have to call ShopifyWrapper::Download.all_products and then deal with the response from it. The wrapper talks to the Shopify APIs.

class ShopifyApi::DownloadAllProductsJob < ActiveJob::Base
  queue_as :default

  around_perform do |job, block|
    shop =  Shop.find_by id: job.arguments.first
    shop.with_lock do
      block.call
    end
  end

  def perform(shop_id, job_created_time)
    shop =  Shop.find_by id: shop_id
    if shop.started_downloading_products == false 
      shop.update(started_downloading_products: true)
      download = ShopifyWrapper::Download.all_products(:shop => shop)
      if download.successful?
        shop.update(sync_time: DateTime.now, downloaded_success: true)
      else
        shop.update(sync_time: DateTime.now, downloaded_success: false)
      end
    end
  end
end

This made testing easier - as I could write Shopify API tests in one place, and test it thoroughly. While writing these ShopifyWrapper tests, there were two different approaches I could take.

I could write tests that cover everything upto hitting the Shopify APIs, and then fake the response coming back from the Shopify APIs.
Or I could fully integrate with the Shopify APIs - and go all the way, and hit their servers in real time and get the real data coming back from the requests. Choosing this option means the wrapper spec will have to handle and interpret the data coming back from Shopify in real time.

For testing the Shopify Wrapper, I chose the second option - i.e to truly integrate with the Shopify APIs and go all the way. For example, the below code shows the RSpec tests for testing the different download methods of the Shopify Wrapper.

require 'rails_helper'

describe ShopifyWrapper do 
  describe ShopifyWrapper::Download do 
    describe ".all_products" do 
      context "with a valid shop" do
        let(:shop) { Fabricate(:shop) } 

        it "returns the products successfully" do 
          response = ShopifyWrapper::Download.all_products(:shop => shop)

          expect(response).to be_successful 
        end

        it "saves the products to the shops database" do 
          shop.products.destroy_all

          response = ShopifyWrapper::Download.all_products(:shop => shop)
          shop.reload

          expect(shop.products.size).to eq(response.response.size)
        end

        it "saves the product images to the shops database" do 
          shop.products.destroy_all

          response = ShopifyWrapper::Download.all_products(:shop => shop)
          shop.reload

          expect(shop.products.first.images.size).to eq(response.response.first.images.size)
        end
      end

      context "with an invalid shop" do
        let(:shop) { Fabricate(:shop, shopify_domain: 'invalidDomain.com', shopify_token: 'invalidToken') } 
        let(:response) { ShopifyWrapper::Download.all_products(:shop => shop) } 

        it "does not return the products" do 
          expect(response).to_not be_successful 
        end

        it "contains an error message" do 
          expect(response.error_message).to eq("There was an error while downloading the products from Shopify.")
        end
      end
    end

    describe ".single_product" do 
      context "with a valid product" do
        let(:shop) { Fabricate(:shop) } 
        let(:product) { shop.products.create(title: "Shirt", shopify_product_id: "4568022220932") } 

        it "it downloads the products successfully" do 
          response = ShopifyWrapper::Download.single_product(:shop => shop, :product => product)

          expect(response).to be_successful 
        end

        it "saves the products images to the database" do 
          product.images.destroy_all

          response = ShopifyWrapper::Download.single_product(:shop => shop, :product => product)
          product.reload
          expect(product.images.first.src).to eq(response.response.images.src)
        end
      end
    end
  end
end

In all other areas of the code which deals with Shopify APIs, I just had to test until my application boundary, and then fake the response from SHopify. I could do this because the ShopifyWrapper spec tests the Shopify APIs thoroughly.

Making Tests Fast with VCRs and cassettes

While running the ShopifyWrapper spec, it takes 9 seconds for actually running the spec.

The reason it is taking so long is that it hits the live Shopify APIs 12 times in this spec. Six times to create a new Shopify session, and six times to download the products from the Shopify store. And as my application grows, I am going to add more and more tests in this spec, and it will take longer to execute.

The only way I could solve this problem and make it faster was to not hit the Shopify API every time - and instead to have a dummy response. I could do this in two ways.

1. Webmocks - This is a library that does exactly this. It stubs the HTTP requests at the http client library level and you can tell exactly what the response is.

For e.g. ruby stub_request(:post, "www.example.com") with(body: "abc", headers: { 'Content-Length' => 3 })

But the problem with this is that I have to change the code in the tests manually. And stubbing the response this way is not ideal - I do want to hit the Shopify API and get back the response from them. What if Shopify changes the way they send data back via their APIs? I wouldn't be able to catch these changes with Webmocks, as Webmocks does this on a low-level. Ideally, I need something at a higher level, and which also integrates with RSpec nicely - so that I don't have to manually change the test code.

2. VCR - This is where VCR is really useful. VCR is a library that records your test's HTTP interactions and replays them during future test runs for faster tests. This is perfect - as it hits the Shopify APIs and gets the response back in real time the first time the test runs. It then records all this information and saves them into a cassette! So, the next time the test runs again, it will just play back the recording - and it will not hit the Shopify endpoints for any subsequent runs.

It is also highly customisable. I can change it to hit the end points all the time if I want to make sure that the Shopify APIs have not changed the way they send data back via their endpoints. Having this option is good because I can switch between these modes as and when required. For the majority of the time, I will be using the VCR recordings - and every once in a while I can hit the end points in real time. This way my tests are going to be secure and run faster.

I added VCR, and you can see that when I run the ShopifyWrapper spec again it takes only 1.06 seconds to run because it did not go to the actual APIs, it just used the local recordings.

I saved 8 seconds from one spec alone by implementing VCR

You can see that VCR automatically created a ShopifyWrapper_Download directory under `spec/cassettes/ShopifyWrapper' and for each context it creates a data file that is uses for the tests. This is super helpful - and it integrated really well with RSpec.

Making Tests Faster with Test Doubles and Method Stubs

In the Test Desiderata videos, Kelly Sutton talks about why speed is important in testing - and also one of the reason they want faster tests is so that they can remain in flow (and switch off the monkey-brain). Paul Graham talks about the makers schedule vs the managers schedule - and how it is important for makers to have dedicated blocks of time just for making.

"For someone on the maker's schedule, having a meeting is like throwing an exception. It doesn't merely cause you to switch from one task to another; it changes the mode in which you work." - Paul Graham

This maybe a bit exaggerated - but I feel slow tests and meetings do exactly the same thing. While I am waiting for the tests to complete, my mind wanders - thinks about lunch, what to buy, plans for the evening, the game last evening - and many other things before I have to re-focus and get it back to programming mode. So, even if I have a dedicated 4-5 hour maker schedule, and no one else is disturbing me - slow tests can interrupt the flow with context switching.

In my application, I talk with the Shopify APIs a lot. And in many different places. I am calling the ShopifyWrapper::Download class itself 14 times in my application. Apart from that, there are other classes in the ShopifyWrapper which are called repeatedly.

Using VCRs does make testing faster. But I can make it even faster by using Test Doubles and method stubs.

This is how the ShopifyWrapper::Download class code looks like.

As I already tested the Shopify Wrapper thoroughly, I don't really need to call the `ShopifyWrapper::Download' class again in other places (14 times!) in my tests. Calling it each time will in-turn call the Shopify APIs (or use VCR recordings). Instead I could stub this method to return something specific - which in this case is going to be a successful download.

So, I started by created a test double object called 'download' --- This is basically a fake object which stands for the actual return value from ShopifyWrapper. I then stubbed the successful method of this class and made it return the 'download' test double we created earlier.

download = double('download')
download.stub(:successful?).and_return(true)

ShopifyWrapper::Download.stub(:all_products).and_return(download)

In another context, when I want to test for an un-successful download, I can again stub this method and make it return false instead. I can also stub the test double with the error message.

download.stub(:error_message).and_return("There was an error connecting to Shopify.")

The reason I am doing this is because I am calling the ShopifyWrapper multiple times in my code. For example, when a new user signs up, and after I create a Shop, the after_commit method calls a job, which in-turn calls the ShopifyWrapper.

class Shop < ActiveRecord::Base
  ...

  after_commit :set_up_shop_first_time

  def set_up_shop_first_time
    ShopifyApi::DownloadAllProductsJob.perform_later(self.id, DateTime.now)
  end

  def api_version
    ShopifyApp.configuration.api_version
  end
end


class ShopifyApi::DownloadAllProductsJob < ActiveJob::Base
  ...

  def perform(shop_id, job_created_time)
    ...
    download = ShopifyWrapper::Download.all_products(:shop => shop)
    if download.successful?
      shop.update(sync_time: DateTime.now, downloaded_success: true)
    else
      shop.update(sync_time: DateTime.now, downloaded_success: false)
    end
  end
end

Now, if I didn't stub the wrapper - it will actually execute the code and hit Shopify apis (or VCR recordings) again. But, because I have stubbed it, it will always return the test double that we created.

This makes my tests faster... I am mainly using Test doubles and stub method for two reasons:

Hitting the Shopify APIs every time is expensive (takes a long time) - so I can replace it with something that is very cheap (Test doubles and stub methods). I first created a test double, and then stubbed the test double itself to make it in a state that I want.
The other reason I am using Test doubles is because the Shopify Wrappers have been thoroughly tested before. And by not repeating the testing in the current spec, it keeps my tests focussed and noise-free.

Speed limits

Even though I wanted my tests to run really fast, I had to slow it down to include feature tests. Feature tests simulate real user experiences - adding new images, re-ordering positions, bulk editing images, and going all the way to the Shopify APIs to update a product with the right image, and then coming back to show the success or the error messages of the response.

For example, here is one feature test - that tests for dragging images between different products, and re-arranging it's position.

In each of my feature tests, I only tested for the happy path. I didn't test for errors or intentionally break things in feature tests. These were already covered in the other tests, which runs much faster than feature tests.

This was a happy compromise I made while writing tests. Even though it made my tests slower, it gave me more confidence in my code.

Do check out the videos from Kent Beck and Kelly Sutton to learn more about the best practices while testing. I like the fun way they break down important concepts into smaller videos. Also, reading the docs from Rspec helped a lot.

Thanks for reading.

Arjun

Integration & Unit Tests with TDD in Rails

Arjun Rajkumar — Wed, 08 Jan 2020 01:36:53 +0000

Hi! I'm Arjun. Currently building ReviewCow

Connect on Github

The advantage of using outside-in TDD is that it flows naturally from the business requirements, to the user actions, and then to the internals of the application - and we don't have to think too far ahead while implementing this. We just have to worry about the next thing in the test, and it will drive you to implement all the details, and then deliver the feature.

The challenge with TDD is that you have to be really familiar with Rails and testing in particular. Because right in the beginning you have to write the high-level integration tests which requires several technologies to work well together. And when the test fails, you have to know exactly what needs to be done to pass the test.

Implementing a new feature via TDD

I have a Shopify app that is live on the app store, and for marketing purposes, I wanted to build a free feature for the app. This post looks into how I went about adding the new feature using Test Driven Development.

This is the sample flow for the new feature:

A user visits the landing page.
He then enters the main keyword and the Shopify product page URL on the form
When he submits the form, the details are saved in the database
A background-job is started in parallel (Lets call it ScrapePageWorker)
The user is taken to the results page

Unit tests are useful for testing business logic in models, and controller tests are useful for testing controllers in isolation - making sure they receive the right inputs and that they render the right response. The above 5 step flow seems ideal for an integration test as we are not dealing with just one controller. Instead we are going all over the application - starting with the landing page, talking to the database, starting a BG job and being redirected to another page.

Integration Tests

I am using Rspec for testing, and I start by creating a new file boostctr/spec/features/free_marketing_tool_spec.rb

Lets start with the first step in sample work flow: A user visits the landing page.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
    end
  end
end

In the next two steps, he enters the details into the form and submits it. I am using Capybara for simulation real-world interactions - to get the virtual user to fill in the details in the form.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
+     fill_in "Enter focus keyword", with: "modular sofa"
+     fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
+     click_button "Check Product Page SEO"
    end
  end
end

Upto now, the user has visited the page, filled in the details and clicked the submit button. At this stage a few things should happen simultaneously - 1) The details are saved to the database, 2) the user is redirected to another page, and 3) a background job starts to scrape the product page to get all the required data.

Let's start with the HTTP side by making sure that the user is redirected to the show page.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
+     expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
    end
  end
end

Next, we'll check the database to make sure that the form details have been saved into a table called Audits, and that the information stored is correct.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
      expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
+     expect(Audit.first.focus_keyword).to eq('modular sofa')
+     expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
    end
  end
end

We also need to make sure that the Sidekiq background job runs properly. For this, we'll call the inline mode.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
      expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
      expect(Audit.first.focus_keyword).to eq('modular sofa')
      expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
+     Sidekiq::Testing.inline! do
+       MarketingTool::ScrapePageWorker.perform_async(Audit.first.id, Audit.first.focus_keyword, Audit.first.url)
+     end
    end
  end
end

Background Workers

Calling Sidekiq inline makes the job run immediately instead of enqueuing it. For this to work, we have to create a Sidekiq background worker, and I do this by creating a new file boostctr/workers/marketing_tool/scrape_page_worker.rb

class MarketingTool::ScrapePageWorker
  include Sidekiq::Worker
  sidekiq_options :queue => 'default', :retry => false

  def perform(audit_id, keyword, url)
    PageScraper.new(audit_id, keyword, url).start_scraping
  end
end

Service Objects

Scraping a page does not really fit into a model or a controller. Also, I feel that I will be using this page scraping functionality in other parts of my app. Therefore, I can use Service Objects for this scenario. Service Objects are Plain Old Ruby Objects(PORO) that are used to execute one single thing, and I can create a new file for this on boostctr/services/page_scraper.rb

For actually scraping and reading the contents of the product page URL I am using the HTTP gem and Nokigiri.

class PageScraper
  attr_reader :audit_id, :keyword, :url

  def initialize(audit_id, keyword, url)
    @audit_id = audit_id
    @keyword = keyword
    @url = url
  end

  def start_scraping
    begin
      response = HTTP.get(url)
    rescue => e
      raise e
    else
      if response != nil 
        code = response.code
        if code && code.to_s.start_with?('2')
          doc = Nokogiri::HTML(response.to_s)
          MarketingTool::CheckTitleTagWorker.perform_async(audit_id, doc.title, keyword)
          MarketingTool::CheckUrlWorker.perform_async(audit_id, get_handle_from_url(url), keyword, keyword)
          MarketingTool::CheckHeaderWorker.perform_async(audit_id, doc.search('h1').map(&:text), keyword)
          MarketingTool::CheckMetaDescriptionTagWorker.perform_async(audit_id, doc.at("meta[name='description']")['content'], keyword) if doc.at("meta[name='description']")
          MarketingTool::CheckKeywordsFirst100Worker.perform_async(audit_id, get_all_image_tags(doc), keyword)
          ...
        end
      end
    end
  end

  def get_handle_from_url(url)
    url.sub(/(\/)+$/,'').match(/\/([^\/]+)\/?$/)[1].gsub("-", " ")
  end

  def get_all_image_tags(doc)
    image_tags = []
    doc.css('img').each do |img_node|
      img_attributes = img_node.attributes.values # list of image attributes
      img_attributes.each do |attr|
        image_tags << [attr.value]
      end
    end
    image_tags
  end
end

The service object above scrapes the page, and gets all the important SEO details of the page. Each SEO test is sent to a different worker. For example, checking if the the page title has a number in it is done by the CheckTitleHasNumberWorker. By separating each task into a different worker, we are ensuring that each worker is small and does only one thing.

class MarketingTool::CheckTitleHasNumberWorker
  include Sidekiq::Worker
  sidekiq_options :queue => 'default'

  def perform(audit_id, title, keyword)
    audit = Audit.find(audit_id)
    title_tag_has_numbers = title.downcase =~ /\d/ ? true : false
    audit.update(title_tag_has_numbers: title_tag_has_numbers)
  end
end

Unit Tests

This completes the whole 5 step user story we wrote in the beginning. But we also need to add Unit tests, to make the Audit model bulletproof - i.e. to make sure that all validations pass before it is saved to the database. I need to test a few validations:

In a Shopify store, all product URLs have to be under the /products folder
The User has to enter both the URL and the focus keyword

For creating a unit test, I add a new file boostctr/spec/models/audit_spec.rb

require 'spec_helper'

describe Audit do 
  it "URL must be a Shopofy product page" do 
    correct_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
    correct_audit.save
    expect(Audit.count).to eq(1)
    wrong_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/sofa')
    wrong_audit.save
    expect(Audit.count).to eq(1)
  end
  it "saves itself" do 
    audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
    audit.save
    expect(Audit.first).to eq(audit)
  end
  it "keyword must not be empty" do 
    audit = Audit.new(url: 'exmaple.com/products/sofa')
    audit.save
    expect(Audit.count).to eq(0)
  end
  it "URL must not be empty" do 
    audit = Audit.new(focus_keyword: 'sofas')
    audit.save
    expect(Audit.count).to eq(0)
  end
end

Conclusion

This post looked at how we can do the following

Using Test Driven Development to build a new feature
Creating integration tests to test the flow across multiple controllers
Adding unit tests to test business logic and validations
Adding background workers to run two or more tasks simultaneously
And adding a Service Object for logic that does not fit into a model or a controller.

How to break Rails built-in Active Record's Validations

Arjun Rajkumar — Fri, 25 Jan 2019 02:15:50 +0000

This post originally appeared on Arjun Rajkumar's blog.

This past week I got an error which as it turn out originated because Rails failed with its built-in Active Record validation checkers.

I have this has_many association set up in my database - a Website has many pages


class Website < ApplicationRecord
  has_many :pages, -> { order("created_at DESC")} , dependent: :destroy
end

class Page < ActiveRecord::Base
  belongs_to :website, touch: true, counter_cache: true
end

Each page has a unique URL scoped to a website. This means that pages under a website shouldn't have the same URLs.

I’m using Sidekiq workers to crawl a website in the background, get all it's on-page links and add those link as pages to the website. Sidekiq is multi-threaded - and runs thousands of jobs at the same time. Each of these jobs is creating a page with the crawled link as the URL. So, it may happen that two different jobs creates two pages with the same URL value at the same time, if it discovers two identical links on the website.

And this is exactly what happened.

Here are a few things I tried that didn’t stop Sidekiq from creating duplicate pages under the same website.

These didn't work.


#Active Record's built-in Validation checkers
class Page < ActiveRecord::Base
  validates_uniqueness_of :url, :scope => :website_id
  belongs_to :website, touch: true, counter_cache: true
end

#Transactions
ActiveRecord::Base.transaction do
  website.pages.create(url: new_url) 
end

This worked!

What worked instead was pretty simple.
Adding a unique index directly in the database solved this error.

class AddIndexToPageUrls < ActiveRecord::Migration[5.2]
  def change
    add_index :pages, [:website_id, :url], unique: true
  end
end

And this works 100% of the time. Rails has documented this properly in their docs, and I fixed the error quickly once I knew what was causing it. But diagnosing this error initially to understand why this is happening took time.

I hope this helps someone in the future.

Write better code: Day 8 - Shortest route

Arjun Rajkumar — Wed, 12 Dec 2018 15:13:53 +0000

Recent articles from Progress Updates:

I found this one interesting...

Problem:

Given information about active users on a network, find the shortest route for a message from one user (the sender) to another (the recipient). Return an array of users that make up this route.

There might be a few shortest delivery routes, all with the same length. For now, let's just return any shortest route.

Your network information takes the form of a hash mapping username strings to an array of other users nearby:

network = {
  'Min'     => ['William', 'Jayden', 'Omar'],
  'William' => ['Min', 'Noam'],
  'Jayden'  => ['Min', 'Amelia', 'Ren', 'Noam'],
  'Ren'     => ['Jayden', 'Omar'],
  'Amelia'  => ['Jayden', 'Adam', 'Miguel'],
  'Adam'    => ['Amelia', 'Miguel', 'Sofia', 'Lucas'],
  'Miguel'  => ['Amelia', 'Adam', 'Liam', 'Nathan'],
  'Noam'    => ['Nathan', 'Jayden', 'William'],
  'Omar'    => ['Ren', 'Min', 'Scott'],
  ...
}

For the network above, a message from Jayden to Adam should have this route:
['Jayden', 'Amelia', 'Adam']

If you want to follow along, feel free to post your answers in the comment.
My answers are in the comments. This problem is from InterviewCake.

Write better code: Day 8 - Graph Coloring

Arjun Rajkumar — Wed, 12 Dec 2018 13:44:02 +0000

Recent articles from Progress Updates:

Day 8: Question 1

Given an undirected graph with maximum degree DD, find a graph coloring using at most D+1 colors.

If you want to follow along, feel free to post your answers in the comment.

My answers are in the comments.
This problem is from InterviewCake.

Write better code: Day 7 - Second largest item in Binary search tree

Arjun Rajkumar — Tue, 11 Dec 2018 08:28:03 +0000

Hi! I'm Arjun. Currently building ReviewCow

Day 7: Question 2

Write a method to find the 2nd largest element in a binary search tree.

A binary search tree is a binary tree in which, for each node, the node's value is greater than all values in the left subtree, and the node's value is less than all values in the right subtree.

If you want to follow along, feel free to post your answers in the comment.

My answers are in the comments.
This problem is from InterviewCake.