The Biggest Gap in QA You've Never Heard Of

The developer ecosystem for test automation is huge. Need to test your web app? Look no further than Selenium or Puppeteer and the myriad frameworks built on top. Is your team only responsible for the API layer? Postman's great for automation and exploration. Oh, your team builds mobile apps? Appium to the rescue.

What few realize is that, despite this vast ecosystem, these tools are all used to maintain quality for just a fraction of the ways in which a customer experiences a product, company, or brand: code. Even at technology companies, code is just one piece of the customer experience pie.

Where No QA Has Gone Before

For example, the process by which a customer discovers, learns about, purchases, and renews a product is almost entirely driven by automation in many organizations. The logic and rules for these business processes are almost exclusively architected, implemented, and maintained by people who are not developers; they are technology administrators in groups like Marketing Operations or Sales Operations.

They automate these processes by purchasing, integrating, and configuring SaaS software like Salesforce, HubSpot, Tealium, and literally thousands more, with only minimal (and commonly no) support from developers or testers.

Some platforms allow business users to define their logic using visual programming.

This shift has been accelerating as subscription-based SaaS software has increasingly become the norm over the past decade. And on the whole, this transition has been good for everyone! Business users are no longer beholden to limited development capacity to solve their problems, meanwhile developers are more free to work on innovation in their products. But there are downsides too.

Far from the QA rigor and influence of product and IT teams, these business technologists end up owning complex, distributed, and brittle systems that break down in emergent and costly ways. What could be worse than a bug in a system with millions of dollars in ads and technology in front of it and the potential for lost revenue and unhappy customers behind it?

Bringing QA to The Rest of the Business

It's with this context in mind that we built a test automation tool called Crank. Our goal is to empower engineers, SDETs, and QA Engineers who support these business users and interface with their technologies to close the quality gap. (And full disclosure: we're also building a business on top of it to empower the business users who don't have developers supporting them at all.)

What is Crank? Well, it's a BDD test automation framework for integrated SaaS. That's a dense summary, so let's unpack it:

BDD: If there was ever a use-case for Behavior-Driven Development, this is it. What better way to collaborate with the business when testing something that they themselves have created? Here's an example Scenario file that works with Crank:

scenario: Contact Form Lead Sync
description: Proves the Contact-Us Form Works.

steps:
- step: Given I navigate to https://example.com/contact?utm_source=ABC
- step: When I fill out input[name=fullName] with Emily Doritos 
- step: And I fill out input[name=email] with [e-mail]
- step: And I submit the form by clicking input[type=submit]
- step: Then the FullName field on Salesforce Lead [e-mail] should be Emily Doritos
- step: And the UtmSource field on Salesforce Lead [e-mail] should be ABC
- step: Finally, delete the [e-mail] Salesforce Lead

Test Automation: Just like developers, business technologists move fast too, so automation is key. Crank is a CLI that can run BDD test scenarios like the one above. You can run it locally before or after a launch. You can install it and run it in your CI/CD pipeline. You can schedule it via cron. Use it however best suits your organization.

Framework: Crank is extensible. It's built on gRPC to enable any developer to write action and assertion steps in any supported language (packaged in docker containers we call Cogs) for any SaaS system. We've built out quite a few Cogs to start, but anyone can contribute to the global library of actions and assertions.

service CogService {
  // Define Cog metadata.
  rpc GetManifest (ManifestRequest) returns (CogManifest) {}

  // Run a single assertion/action step.
  rpc RunStep (RunStepRequest) returns (RunStepResponse) {}

  // Run a series of assertion/action steps.
  rpc RunSteps (stream RunStepRequest) returns (stream RunStepResponse) {}
}

For Integrated SaaS: By this, we mean two things.

First, Crank is not a test automation framework for everything; you won't use it to test your product's core features, just where your product or site interfaces with other SaaS technology. Limiting its scope in this way allows us to stay focused in an already immense problem space.

Second, Crank's action and assertion steps rely on the APIs of SaaS platforms. The Salesforce Cog, for example, exposes steps that use Salesforce CRUD APIs to alter and inspect the state of a Salesforce instance. When you author a Scenario that spans multiple platforms, you can start to see the power in testing the complex ways in which SaaS tools interact.

---

Think this tool might help? Install Crank Now.

Don't really work with this kind of stuff? Forward this article to a colleague who does! Better yet, seek out those marketing and sales technologists in your company and help them! They need it.