DEV Community: Stella Crowhurst

Our adaptation of shift-left in QA

Stella Crowhurst — Fri, 06 Nov 2020 09:00:50 +0000

It’s almost a parallel universe. Although part of a global organization with vast reach and huge responsibilities, Access Worldpay remains a small, autonomous unit with the freedom to choose its own working principles and methods. Not only can it adopt best practices, but it can also hack these as necessary to meet its needs. Co-lead Build Chapter, Pat Bateman explains how his team’s approach to QA demonstrates its independent, entrepreneurial spirit.

The starting point, he says, are the objectives. Access Worldpay is expected to turn work around fast and to a consistently high standard. “Predictability is one of the big things here. We want to get stuff out of the door really quickly and if there’s no predictability, then the business has a difficult time working with a customer.”

Goodbye Waterfall

Such imperatives immediately made the team determined to streamline its working processes. Quality assurance (QA) was a prime candidate for reform. Standard practice separated the QA function from software development. “They were almost two different organizations,” says Pat. “Software developers would write the code; when it was finished, they’d give the application to the test department for validation.”

This approach made the interplay between QA and development slow and inflexible. The long feedback cycles in the Waterfall approach to software development also made it expensive. “You’d have these massive cycles to find out something was wrong and the expense of going back and fixing the issue was immense,” Pat says. “And often what you thought you were building, in terms of set requirements, had changed.”

Improving Agile

The shorter cycles within the Agile approach helped a little, but not enough, so the team explored further refinements. “Even in Agile, the QA function remained similar to the old methodology of ‘test after’,” says Pat. “What we’ve discovered is that you can make more change the earlier in the product’s life-cycle you go.”

This is ‘shift-left’: moving the quality assurance to the left – to the start of the cycle. It means that QAs can work alongside the product owner and the developer to help ensure that the requirement is correct and agree on the acceptance criteria. “This makes our whole life-cycle more efficient because you’ve done much more detection of fault or quality at the beginning of the requirement description,” says Pat. “So by the time the developer starts coding, they’re going to find fewer problems with the requirement.”

Expanding the role

The QA function also extends to exploratory testing. “QAs come up with extra hypotheses, also at the beginning, thinking about the other scenarios we haven’t documented,” says Pat. “They’ll ask ‘does it work in this situation?’ and they may find some edge cases we hadn’t thought about, so they’re coming up with some requirements we’d missed.”

And, name aside, shift-left does not confine QAs to the start of the cycle. They still perform their traditional role at the other end, confirming that the software engineers have met the requirement and fulfilled the acceptance criteria. In practice, therefore, QAs are now available throughout the cycle, helping software engineers as needed to ensure that they are interpreting requirements and acceptance criteria correctly. “The QA becomes a more important person in the delivery squad,” says Pat: “They’re constantly being used.”

Widening the search

Although shift-left has improved the team’s productivity, the shake-up it involves can make recruitment somewhat tricky. For example, software developers are now expected to automate all their tests, whereas beforehand they shared that responsibility. And Pat recognizes that traditional QAs find shift-left to be “a difficult transition.” He continues: “We are essentially asking them to change their job – to move much more to a product view of the world. It’s very difficult to find people who understand it thoroughly.”

To mitigate this challenge, new recruits receive “lots of support on the job,” he says. Access Worldpay bring in QAs with shift-left experience to help new colleagues at the initial, refinement stage – “they help them go through exemplar requirements and acceptance criteria, and we iterate this to keep helping them improve.” QAs and product owners also have a two-day workshop, where external experts go through the whole process and teach them how to find out whether what requirements are viable.

“A bigger impact”

Pat is convinced that this extra effort is very worthwhile, not least for QAs themselves. “Traditionally, the QA function has been at the wrong end of the process,” he says. “You can feel like you’re an afterthought, pushed into very tight timelines because earlier parts of the process have run over, so you’re just sat there, checking stuff and filing bugs without any feel of the process.” By contrast, he says, shift-left “places you at the front, with a bigger impact on the product definition; it makes you one of the most crucial people within the product you’re trying to deliver. For QAs right now, this is an exciting time.”

“If it’s not tested, it’s broken” – how we test infrastructure code

Stella Crowhurst — Fri, 30 Oct 2020 12:54:25 +0000

Principles matter. For any group, of any size, principles bring clarity, they provide a sense of purpose and they offer direction when the road ahead is difficult or complex.This is certainly true at Access Worldpay team, where test-driven development (TDD) is a guiding principle. Does it still hold true when put under stress?

At Access Worldpay everything is tested, all the time, and the phrase “If it’s not tested, it’s broken” has become a mantra for the team. Rigorously applying this principle takes effort, even in software development where TDD has become commonplace. Maintaining the same rigour becomes far more challenging in newer areas, particularly infrastructure code. Why?

Big change, new skills
Infrastructure itself has changed rapidly and significantly, from localized hardware and software to remotely-located cloud-hosted services, such as AWS. This requires users to configure and code remote servers to their precise needs and then test that this code meets their required specifications.

Testing infrastructure code is, therefore, an entirely new discipline – one that is wholly embraced by Access Worldpay, albeit with open eyes. Dominic Byrne, a cloud engineer, explains: “Infrastructure code at the moment is in its infancy – for example, terraforming is releasing at a pretty rapid rate. It’s what we use, but it’s still in beta.”

New code, no context
As a result, Dominic explains, reliable testing frameworks do not yet exist. “People are reluctant to develop them because early ones don’t work anymore,” he says. “They were just built around a basic configuration plan so they become completely useless whenever the structure of that plan changes.”

What’s more, many cloud engineers are unused to TDD. “In the past, infrastructure engineers and application developers were two different teams,” says Daniel Beddoe, a senior software engineer at Access Worldpay. In addition, he explains, people used to question why infrastructure configuration files needed testing at all. “It turns out there’s every need to do so because now you could write something very bad in just one line of configuration that could mess up your whole infrastructure.”

Same team, single path
Instead of treating application and infrastructure engineering separately, Access Worldpay takes the same approach to every piece of work. “We’re making everything one team and applying practices from application development into infrastructure,” says Dan. This means identifying requirements and defining success before work begins on each project, whatever its nature: “We’re saying that you need to write tests for your infrastructure just as much as you need them for your applications.”

Part of this shift is organic. “People are increasingly moving from application development into operations and infrastructure,” Dominic explains. “They like the tools they currently use and they want to bring the same principles into infrastructure code.”

Live tests, late feedback
That’s easier said than done, because infrastructure code is tested against a live service, which makes the feedback loop much longer. “Without building anything, loading my spec files and running my tests on my current project averages about 44 seconds,” says Dominic: “For an application developer, tests taking that long to finish aren’t good.”

A central goal, therefore, is to ensure that this feedback cycle remains manageable – particularly since it lengthens further whenever code is altered. “If I change, say, an elastic search server, the loop goes from 44 seconds to 10 minutes at least – that’s just the reality of those services online,” Dominic explains. “So for infrastructure code, following basic TDD principles where you write your tests and watch them fail means allowing for the time it takes in your feedback loop.” That this can be difficult is hardly a surprise given the pioneering nature of the work, but the team has devised new ways to ameliorate these challenges.

Testing spec, teaming up
One approach is to change how infrastructure code is written before testing begins. Instead of writing and testing segments of code incrementally and enduring the long feedback loop each time, “you try to write each instance as the final product,” Dominic explains. “For example, saying ‘I want it to have these forwarding rules; I want this origin to match this specification,’ and you write all that and then you start your feedback loop to test that spec.”

The issue with working this way, of course, is unpicking the code should a test fail. But perhaps such workarounds need only be temporary; as infrastructure code is more widely adopted and tested, eventually more robust frameworks to support those tests may be built

In the meantime, Access Worldpay is supporting TDD for infrastructure code by physically pairing software developers and cloud engineers to shed old, silo thinking and encourage the broader outlook that cloud-based infrastructure requires. “There’s a whole world of delivery to actually get your thing to work – you need to know how to get your application out into the world,” Dominic says: “Just knowing how it works locally isn’t useful for anyone.”

Towards better, even slowly
However imperfect today’s situation, Access Worldpay remains committed to testing infrastructure code. Partly this is to uphold the principle that TDD produces better, more reliable products for its customers, and partly because testing documents systems more clearly. This makes it easier for product owners to prioritize future work and for new team-mates to make useful changes quickly and with confidence.

The end goal of being able to apply traditional application development practices to infrastructure code in a consistent, rigorous way is still some way off. In the meantime, Access Worldpay is trying to develop a methodology that makes infrastructure code tests not only rigorous but also practical – staying true to its core principle while always analyzing its approach and making further refinements wherever necessary or possible. Progress, however uncertain, is still progress.