Recent Articles:
-
The advantage of using outside-in TDD is that it flows naturally from the business requirements, to the user actions, and then to the internals of the application - and we don't have to think too far ahead while implementing this. We just have to worry about the next thing in the test, and it will drive you to implement all the details, and then deliver the feature.
The challenge with TDD is that you have to be really familiar with Rails and testing in particular. Because right in the beginning you have to write the high-level integration tests which requires several technologies to work well together. And when the test fails, you have to know exactly what needs to be done to pass the test.
Implementing a new feature via TDD
I have a Shopify app that is live on the app store, and for marketing purposes, I wanted to build a free feature for the app. This post looks into how I went about adding the new feature using Test Driven Development.
This is the sample flow for the new feature:
- A user visits the landing page.
- He then enters the main keyword and the Shopify product page URL on the form
- When he submits the form, the details are saved in the database
- A background-job is started in parallel (Lets call it ScrapePageWorker)
- The user is taken to the results page
Unit tests are useful for testing business logic in models, and controller tests are useful for testing controllers in isolation - making sure they receive the right inputs and that they render the right response. The above 5 step flow seems ideal for an integration test as we are not dealing with just one controller. Instead we are going all over the application - starting with the landing page, talking to the database, starting a BG job and being redirected to another page.
Integration Tests
I am using Rspec for testing, and I start by creating a new file boostctr/spec/features/free_marketing_tool_spec.rb
Lets start with the first step in sample work flow: A user visits the landing page.
require 'spec_helper'
feature "free marketing tool" do
scenario "first time" do
visit audits_path
expect(page).to have_content("Shopify Product Page SEO Checker")
end
end
end
In the next two steps, he enters the details into the form and submits it. I am using Capybara for simulation real-world interactions - to get the virtual user to fill in the details in the form.
require 'spec_helper'
feature "free marketing tool" do
scenario "first time" do
visit audits_path
expect(page).to have_content("Shopify Product Page SEO Checker")
+ fill_in "Enter focus keyword", with: "modular sofa"
+ fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
+ click_button "Check Product Page SEO"
end
end
end
Upto now, the user has visited the page, filled in the details and clicked the submit button. At this stage a few things should happen simultaneously - 1) The details are saved to the database, 2) the user is redirected to another page, and 3) a background job starts to scrape the product page to get all the required data.
Let's start with the HTTP side by making sure that the user is redirected to the show page.
require 'spec_helper'
feature "free marketing tool" do
scenario "first time" do
visit audits_path
expect(page).to have_content("Shopify Product Page SEO Checker")
fill_in "Enter focus keyword", with: "modular sofa"
fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
click_button "Check Product Page SEO"
+ expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
end
end
end
Next, we'll check the database to make sure that the form details have been saved into a table called Audits, and that the information stored is correct.
require 'spec_helper'
feature "free marketing tool" do
scenario "first time" do
visit audits_path
expect(page).to have_content("Shopify Product Page SEO Checker")
fill_in "Enter focus keyword", with: "modular sofa"
fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
click_button "Check Product Page SEO"
expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
+ expect(Audit.first.focus_keyword).to eq('modular sofa')
+ expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
end
end
end
We also need to make sure that the Sidekiq background job runs properly. For this, we'll call the inline mode.
require 'spec_helper'
feature "free marketing tool" do
scenario "first time" do
visit audits_path
expect(page).to have_content("Shopify Product Page SEO Checker")
fill_in "Enter focus keyword", with: "modular sofa"
fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
click_button "Check Product Page SEO"
expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
expect(Audit.first.focus_keyword).to eq('modular sofa')
expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
+ Sidekiq::Testing.inline! do
+ MarketingTool::ScrapePageWorker.perform_async(Audit.first.id, Audit.first.focus_keyword, Audit.first.url)
+ end
end
end
end
Background Workers
Calling Sidekiq inline makes the job run immediately instead of enqueuing it. For this to work, we have to create a Sidekiq background worker, and I do this by creating a new file boostctr/workers/marketing_tool/scrape_page_worker.rb
class MarketingTool::ScrapePageWorker
include Sidekiq::Worker
sidekiq_options :queue => 'default', :retry => false
def perform(audit_id, keyword, url)
PageScraper.new(audit_id, keyword, url).start_scraping
end
end
Service Objects
Scraping a page does not really fit into a model or a controller. Also, I feel that I will be using this page scraping functionality in other parts of my app. Therefore, I can use Service Objects for this scenario. Service Objects are Plain Old Ruby Objects(PORO) that are used to execute one single thing, and I can create a new file for this on boostctr/services/page_scraper.rb
For actually scraping and reading the contents of the product page URL I am using the HTTP gem and Nokigiri.
class PageScraper
attr_reader :audit_id, :keyword, :url
def initialize(audit_id, keyword, url)
@audit_id = audit_id
@keyword = keyword
@url = url
end
def start_scraping
begin
response = HTTP.get(url)
rescue => e
raise e
else
if response != nil
code = response.code
if code && code.to_s.start_with?('2')
doc = Nokogiri::HTML(response.to_s)
MarketingTool::CheckTitleTagWorker.perform_async(audit_id, doc.title, keyword)
MarketingTool::CheckUrlWorker.perform_async(audit_id, get_handle_from_url(url), keyword, keyword)
MarketingTool::CheckHeaderWorker.perform_async(audit_id, doc.search('h1').map(&:text), keyword)
MarketingTool::CheckMetaDescriptionTagWorker.perform_async(audit_id, doc.at("meta[name='description']")['content'], keyword) if doc.at("meta[name='description']")
MarketingTool::CheckKeywordsFirst100Worker.perform_async(audit_id, get_all_image_tags(doc), keyword)
...
end
end
end
end
def get_handle_from_url(url)
url.sub(/(\/)+$/,'').match(/\/([^\/]+)\/?$/)[1].gsub("-", " ")
end
def get_all_image_tags(doc)
image_tags = []
doc.css('img').each do |img_node|
img_attributes = img_node.attributes.values # list of image attributes
img_attributes.each do |attr|
image_tags << [attr.value]
end
end
image_tags
end
end
The service object above scrapes the page, and gets all the important SEO details of the page. Each SEO test is sent to a different worker. For example, checking if the the page title has a number in it is done by the CheckTitleHasNumberWorker. By separating each task into a different worker, we are ensuring that each worker is small and does only one thing.
class MarketingTool::CheckTitleHasNumberWorker
include Sidekiq::Worker
sidekiq_options :queue => 'default'
def perform(audit_id, title, keyword)
audit = Audit.find(audit_id)
title_tag_has_numbers = title.downcase =~ /\d/ ? true : false
audit.update(title_tag_has_numbers: title_tag_has_numbers)
end
end
Unit Tests
This completes the whole 5 step user story we wrote in the beginning. But we also need to add Unit tests, to make the Audit model bulletproof - i.e. to make sure that all validations pass before it is saved to the database. I need to test a few validations:
- In a Shopify store, all product URLs have to be under the /products folder
- The User has to enter both the URL and the focus keyword
For creating a unit test, I add a new file boostctr/spec/models/audit_spec.rb
require 'spec_helper'
describe Audit do
it "URL must be a Shopofy product page" do
correct_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
correct_audit.save
expect(Audit.count).to eq(1)
wrong_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/sofa')
wrong_audit.save
expect(Audit.count).to eq(1)
end
it "saves itself" do
audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
audit.save
expect(Audit.first).to eq(audit)
end
it "keyword must not be empty" do
audit = Audit.new(url: 'exmaple.com/products/sofa')
audit.save
expect(Audit.count).to eq(0)
end
it "URL must not be empty" do
audit = Audit.new(focus_keyword: 'sofas')
audit.save
expect(Audit.count).to eq(0)
end
end
Conclusion
This post looked at how we can do the following
- Using Test Driven Development to build a new feature
- Creating integration tests to test the flow across multiple controllers
- Adding unit tests to test business logic and validations
- Adding background workers to run two or more tasks simultaneously
- And adding a Service Object for logic that does not fit into a model or a controller.
Top comments (0)