DEV Community: Dan Makarov

Shift Left Testing

Dan Makarov — Tue, 20 Jul 2021 17:46:26 +0000

Shift left testing approach is a popular technique in the QA industry. The approach was coined by Larry Smith in 2001 and has its roots in agile software development. The idea behind shift left testing is to reduce defects earlier and more often so that they are easier to fix, less costly, and less stressful on developers. It answers the problem of accelerating software development without compromising on quality. But what does it mean?

The article discusses the benefits of shift left testing in software development life cycle and key considerations for how a product's success can be maximized by utilizing this approach.

Shift Left Testing Approach

If you look at the traditional software development models, testing enters the picture at the end of the pipeline, or simply put, at the extreme right side of the pipeline.

Even today, most teams feel like testing is the last step in the development cycle. But bringing testing at the very end gives rise to many problems, such as delay in release cycles, low-quality products, or increased development and testing costs. The shift left testing approach pushes testing to the left, or to the earlier stages in the pipeline. This enables teams to find and resolve bugs as early as possible in the development process.

Shift left testing not only increases the collaboration between testers and developers, but it also makes identifying key aspects that need to be tested early in development a whole lot easier

Benefits of Shift Left Testing

Increased delivery speed

It's not rocket science that the sooner you start, the sooner you finish. If you find critical bugs early in the Software Development Cycle, you can fix them earlier and more efficiently. This helps in significantly reducing the time between releases, and with that, increases the delivery speed.

Enhanced test coverage

When you start the test execution right from the origin of the development process, the software can be quickly evaluated or tested for all the features, functionalities, and performance. Shift left testing naturally increases the test coverage percentage. Increased test coverage has a significant impact on the overall quality of the software.

Streamlined workflow

Shifting left is hard to implement—but worth the time and effort. It gives the QA team enough time to go deeper into a product and implement innovative testing solutions. Moreover, it helps the testing team in getting more comfortable with the tools and technology concerned. Additionally, shift left testing makes it easier to streamline several aspects of the software development process.

Example
Testing early facilitates a stronger bond between business requirements and QA scripts, motivating the team to automate more test scripts as part of the requirements gathering process.

Reduced development and testing cost

Debugging is one of the most challenging aspects of software development. As a general rule, the cost of fixing an issue increases significantly as the software moves forward in the SDLC. The sooner you find your mistakes, the easier they will be resolved.

Consider an example of a payments app finding a security flaw after releasing its latest app version. Sure, there still would have been some cost to fix if their team had found the issue earlier in development. But now, the company will spend significantly more time, effort, and money to fix it. The complexity of implementing changes in a live production environment really does make it hard to do anything late-stage, not to mention the associated overall costs that come with late stage maintenance.

The cost of fixing bugs in the real world

Gartner has estimated the cost of network downtime to be $5,600 per minute---totaling to over $300,000 per hour. A real-world example of catching a bug in production is Microsoft Azure's leap-year bug that left some customers in the dark for more than 12 hours. The outage was caused by a software bug triggered by the Feb. 29 leap-year date that prevented systems from calculating the correct time.

Improves product quality

Shift left testing approach positively impacts the overall code quality with stringent and frequent code quality checks. It facilitates timely correspondence between stakeholders, developers, testers, and timely feedback, which helps improve the code quality. This ensures that your customers will get a stable and high-quality end product.

Best practices for shift left testing

Proper planning: Once the product requirements have been finalized, developers and testers should plan how to shape coding and testing as early as possible in the development cycle. Simply put, shape the development in such a way that it remains friendly to shift left testing.
Complete understanding of product requirements: It is important to make sure that your testing team has deep know-how of the product requirements and actively takes part in all the team meetings to clarify those requirements.
Offer continuous feedback: Developers and testers should be offered timely feedback throughout the software development lifecycle. This helps avoid the unwanted gaps across the teams and helps everyone get involved much better.
Specify quality standards: As developers are not trained in testing from the ground level, QA managers should outline the quality standards so that the developers who are running tests have clarity on the type of bugs to look for.
Embrace test automation: The development team should leverage test automation as shift left testing involves frequent testing. It keeps the pressure off the testing team in providing quick feedback about the stability of the code and checking whether all the functionalities are working as expected or not. Automation drives shift left testing in a big way and ensures minimal bugs are found later on in the software development life cycle.

It makes sense to have a tester work on a feature from the very beginning, and they should get involved in all the project meetings, right from requirement gathering meetings to review code meetings.

The agile testing pyramid

Created by Mike Cohn, the agile testing pyramid is a very important model in Shift Left testing---they usually go hand in hand. This model helps testers understand how to implement automated testing. The bottom part of the pyramid represents types of testing that benefit most from automation. As you move up the hierarchy, exploratory testing---on the top of the pyramid--- is non-automated.

The pyramid acts as a 'single source of truth' for QA testers to refer back to and meet various needs. Yet, there are still chances that bugs are missing because the agile testing pyramid might not cover edge cases.

Conclusion

The shift left approach enables the QA team to detect bugs sooner and the development team to fix them quickly before they impact the end-user. This approach not only amplifies the quality throughout the entire software development life cycle but also significantly reduces the overall costs involved and accelerates the release cycles.

It's time to embrace shift left methodology and give your product development a significant push!

What we learnt from 3000 bug reports

Dan Makarov — Tue, 03 Dec 2019 09:54:39 +0000

When do people report bugs? You probably never asked yourself that question, which was one of the reasons we wanted to answer it. The second reason was that we had the right type of data available, since our tool helps teams report and fix bugs. Long story short, we decided to learn more about the way software development teams approach the bug reporting process. For that exercise we've taken a random sample of 3000 bug recordings uploaded to our database in the last 2 months. Here's what we found.

2pm is the golden hour for reporting bugs

Turns out that 2pm is the golden hour for reporting bugs. It is interesting to see that activity is growing in the morning up to 9am, and then again towards the afternoon, peaking at 2pm. There is also a local spike at 5. It's hard to know for sure why that is. A possible explanation is that some people do a round of testing before the end of their work day. That brings us to the next point - working hours.

Is there even such a thing as a standard working day in software development circles? Turns out - not really. We decided to compare how many recordings are uploaded within the regular 9-to-5 as opposed to other times.

The split is almost equal, which was at first a bit surprising to us. When you look at the previous graph (bugs reported by hour), there seems to be more activity during the normal working day. You therefore don't expect an almost perfect 50/50 distribution. The explanation is quite straightforward though - 9-to-5 is only 1/3 of the day. So the remaining period is twice as long in terms of hours, but the rate of reporting bugs is two times lower. That calls for a question: are the individuals in product teams more flexible these days? Or are we just working longer and should reconsider our habits?

A quarter of bug reports are filed on Tuesdays

We expected people to file fewer bug reports on Fridays and weekends. This assumption proved to be correct. What we didn't expect is that the first 2 days of the week would see higher activity. Our assumption was that there would be little to no difference between Monday, Tuesday, Wednesday and Thursday.

2/3 of bugs are reported on pre-production environments

According to a 2009 study by Capers Jones, 85% of bugs are caught before the code gets to production. Is that the case today?

In our particular sample about 2/3 of bugs are reported on pre-production environments (local and staging). That number is lower than the one in the original research by Capers Jones.

One of the plausible reasons can be a different user group. While Capers Jones seems to have focused primarily on software development teams, our product is used not only by Engineers, QA, PMs and Designers. We see an increasing amount of activity from customer support teams, for example. That means 2 things. First of all, business and operations roles in general don't use staging, so they can only report bugs on production. Secondly, certain things they might report as bugs, aren't actually bugs. Even a person, who knows the product inside-out, can't immediately tell if something's indeed a bug every single time. Some behaviours can be caused by hard-to-find settings, ad blockers or connection issues.

We can't say for sure if this assumption correctly explains the 20% difference in the number of bugs reported on pre-production environments. Many factors are at play: company size, development methodology (agile/waterfall), quality assurance process, complexity of business logic and even what a particular company defines as a bug.

"Bug" is the most common word in bug reports

For that exercise we used the titles of bug reports to create a word cloud and see the most common themes.

The result turned out to be rather predictable. Words like Bug, Error, Page, Test were the most common. To be frank, we did not expect "Collection" and "Date" to be that frequent - could be due to the sample, though.

As the next step we removed the 7 highest ranking words to understand the rest of the group better. Now it's more apparent that adjectives and verbs are also on the list. Despite that, majority of the words are nouns.

What we found the most interesting, and in a very positive way, is what words were NOT on top of the list. From personal experience we know that sometimes people describe bugs in very general terms, i.e. "Nothing works", "X is broken". To help companies avoid that, we have even created a bug report template to help especially non-technical people communicate bugs better. Yet, in our sample that behaviour was gladly uncommon.

Most bug reports are made with macOS

Despite having only 17% of the general market, macOS is the most popular choice among bug reporters. More bug reports were made using this operating system, than Windows and Linux combined. Could it be because macOS makes you more productive?

The majority of usage in the last 2 months came from the previous version of macOS (Mojave). Lately though we see more activity from Catalina, as you might expect.

The data we have on Windows is a lot less granular. We can only say that almost all recordings were uploaded using Windows 10.

Google keeps Chrome users on the latest software

At the moment our product is officially supported only on Chrome, which is why we decided to exclude information about other browsers. We can however say that Chrome accounts for over 85% of our website traffic.

The earliest bug reports in our dataset are 2 months old. At the same time, very few of them came from Chrome versions released over 4 months ago. It seems that Google is fairly good in making sure that users upgrade to newer versions.

As a company focused on bug reporting, we can only recommend this strategy as a best practice - it makes things much easier to debug. It also ensures a more homogeneous user base, which has positive implications not only on software development teams, but also technical support.

Bug error. Test collection date issue.

To conclude, we decided to create a typical bug report based on the data we had. To do that, we combined the information from this article with the figures that didn't make it into the graphs above.

And here it is - the most "average" bug report of this fall.

Title: Bug error. Test collection date issue.
Time: Tuesday, 2pm
Environment: Staging
Operating system: macOS 10.14.6
Device vendor: Apple
Browser: Chrome 77
Screen size: 1920x1080
Window size: 1920x798
Locale: en-US

How to report bugs (if you want them fixed)

Dan Makarov — Mon, 16 Sep 2019 20:23:51 +0000

Intro

Most of the time engineers get poor bug reports that they cannot act upon. It happens because "an average person" is usually not familiar with what information can help engineers fix the issue faster.

We tried to create a guide that would be easy for anyone to understand. Feel free to send around the information below, especially when you get a bug report with little useful information.

Bug report template

Title
Steps to reproduce*
Resulting and expected behaviour
Screen recording or screenshot
System information*
- Operating system
- Browser
- Device
- Time (including time zone)
User ID [If relevant]
Impact [If known]

*Some screen recording tools include this information automatically (see more below).

How to use this template

The most important thing to understand when reporting a bug is that the reader should be able to reproduce it on their computer only with the information provided. It is not always possible to reproduce the bug, but it's best to provide as much information as possible.

Title

Briefly describe the issue, i.e. "Calculator: incorrect result of multiplication". Try to be concise, but specific here. For example, "Broken landing page" doesn't help the reader to understand whether the issue has already been reported and what exactly is wrong: Is it not accessible? Is the layout broken? Is it just a typo in one of the headlines?

Steps to reproduce

Describe what actions you took before you encountered a bug. It can look something like this:

Go to ExampleURL.com
Click button A
Select option B
Type "123"
Click Enter

It is best to try to reproduce the bug yourself, so that you can be sure that the steps provided will lead to a bug you are reporting.

Resulting and expected behaviour

Specify what the outcome of your actions was compared to what you would expect the outcome to be. Examples:

Thing X should happen. In reality thing Y happens
I get an error "404. Page not found."

Screen recording or screenshot

There are many screen recording tools for you to choose from. Mac users can use QuickTime (see tutorial here) or by pressing Shift-Command-5. On Windows 10 you can use the Xbox app (see tutorial here).

A faster way to make a screen recording for a bug report is to use a dedicated tool (i.e. Bird or BugReplay). Together with the video it automatically captures technical information that will help engineers fix the problem a lot quicker. This way you also don't need to worry about providing system information and typing up the steps to reproduce the bug - they will be documented automatically as well.

In certain cases (i.e. typo on a landing page) there's no need to record a video - a screenshot is sufficient. To make one you can use the tools built into your operating system. See how how to make a screenshot on a Mac here, Windows - here.

System information

Operating system (including version): Click here to find out this information.

Browser (including version): Click here to find out this information.

Device: Desktop / Tablet / Mobile

Time (including time zone): At least approximate time when you encountered a bug.

User ID [If relevant]

If you or the reporter of the bug were logged in when the bug happened, provide the unique identifier of the account. It is usually either username or e-mail.

Impact [If known]

In case you know how many people are affected by the bug or how much revenue is being lost because of the bug, definitely provide this information. It will help engineers and product managers assign the correct priority.

Final thoughts

If you want your or your customer's issue to be resolved quickly, please try to file a good bug report - the one that makes it easy to reproduce the bug with the information provided. The more information engineers have, the faster they are able to implement a fix.

It's not uncommon for product teams to receive like the ones below:

Nothing works
Page doesn't load
Everything crashes
I don't see anything
X is broken
etc.

Please, avoid those at all costs. It will help both you and people, fixing the bug, to save time and frustration.

Does it ever make sense to estimate bugs?

Dan Makarov — Thu, 01 Aug 2019 05:36:16 +0000

Should we estimate bugs? Countless hours have been spent by teams all over the world debating over this. But really, should you?

Nobody can answer this for you, but let’s raise the question anyway and prepare for a bunch of contentious comments. So what are the options for estimation we have in the first place?

Dedicated time for bug fixing

Instead of estimating bugs beforehand, some teams just set aside some time each sprint / day / week / month for bug fixing. It doesn’t mean though that they never estimate bugs. If after the initial investigation it turns out to be a bigger fix, or it requires a change to the behaviour of the product, they would still estimate it, but most likely treat it almost like a feature, that might, if needed, go through the complete process of specification, design, development, testing, release.

Default estimation

Another way to approach bug estimation is to use some placeholder, like 0.5–1 days, for every bug. That idea actually comes from Jeff Sutherland, one of the creators of Scrum. These placeholders can be rather precise as most bugs don’t take more than a day to fix — Steve McConnell in his (by now rather old) book Code Complete mentions that 85% of bugs take less than a few hours to fix.

We’ve also seen certain teams take it to an extreme and treat all tickets this way. The reasoning behind it is that over a long enough period of time not only do the things average out, but also the people learn to work better together and the tickets created become roughly the same size.

Irregardless of whether you want to use this technique just for bugs or features as well, you can start with 0.5 day placeholder and adjust it as you get more data on how long it actually takes developers to resolve issues. Retrospectives are a good way to keep track of that.
If you want to get one step further and have more granular estimations, you can divide bugs by type, i.e. coding bug, design bug, database, business logic, state machine, etc. That way you would be able to have a different placeholder per type and iterate them individually.
The more data you have, the better estimations you would get, which brings us to the next approach.

Estimation with historical data

If you have enough data, cool things can be done with estimations — allowing for much more granularity than splitting issues by type and assigning individual placeholder, like we’ve seen in the previous example. You could create a much more contextual system, that would use historical data from your issue tracker (Jira, Trello, etc.) to predict how long a certain fix would take using Machine Learning, Natural Language Processing and other approaches.

In fact, a group of researches all the way back in 2007 did just that. They came up with a system that automatically predicts the fixing effort. Given a sufficient number of issue reports, their automatic predictions for bugs were off by only one hour, beating human predictions by a factor of four. You can read more about it here.

No estimation

The last school of thought we will touch upon believes that since you can’t estimate how long it will take to fix a bug until you’ve located the underlying problem, and most of the time is spent on precisely that, trying to come up with an estimate is pointless.

So should you estimate bugs?

There is no “one-size-fits-all” solution here unfortunately, which won’t stop us from trying to give an answer to the question.
There are many things that should influence your decision on whether to estimate bugs and if so, which approach to use. To give an example: A bank would handle things differently than a social network, culture and motivation in one company or country may influence the decision, even the stakeholders matter — if you have a project manager who cannot live without estimations, it might make sense to just go along with it and have a good team atmosphere rather than continuously fight over estimations.

The biggest influence on whether you should estimate bugs or not, is the size of your team and company, as these two factors influence a lot of other things, so let’s take a look at a couple of different setups and what you might want to do in each case.

Solo founder

You are building a small product or just a personal project. There are no dependencies on other people, precise planning is not really necessary because you know how fast you can progress and what you need to work on next.

Solution: It’s likely that you are not doing formal estimations for features, so no need to estimate bugs either. Just focus on output and getting that code out there!

Small company

You have a 5–10 person development team and no other engineering teams depend on it. You are planning sprints and tracking velocity to roughly understand how good your progress is. Some bugs are caught by people internally, some — reported by users.

Solution: In order to plan your sprint capacity better, you should give default estimations of half a day for bugs (unless you have more information about a specific defect and can estimate more precisely), but also — don’t plan 100% of capacity for the sprint, leave some buffer — it will allow you to a) be prepared for some bug fixes to take longer and more importantly b) be always ready to quickly fix an important unforeseen bug without sacrificing sprint goals.

We have built a few early-stage products before, and over time came to this setup. When the product is new, you don’t have that much historical data to base your decisions on, don’t really have a proper QA team, so you need to be nimble and ready to react to immediate needs while still maintaining direction and communicating your goals and progress to other teams (sales, marketing, etc.). This setup also helps with building good customer relationships, as users are always impressed when you are able to fix something soon after their report.

The longer you work on the product, the better you are able to estimate, of course. Especially if you split bugs by type, i.e. backend, frontend, etc., so the default estimations keep getting better.

Large company

You have a few development teams and at least some of them depend on each other. You have manual QA and QA engineers, but very limited ability to talk directly with a user. Lots of stakeholders and multiple layers of management exist, to whom you need to communicate your goals, so you really need to make sure you deliver on your promises. In general, you care much more about reliability of the software and predictability of development output rather than speed and agility.

Solution: Estimate all the bugs you can. Start with at least an educated guess, but try to use historical data for better predictions. In a large organisations an effort to build such a prediction model will not be in vain — other teams will use it and, if it works well, it can save time during planning meetings, but also make the velocity much more predictable, which your management will appreciate.

To conclude

In many cases at least one engineer knows exactly what the source of the bug is, and exactly how to fix it. There is no reason not to estimate those. On the other hand, some bugs are totally obscure and unpredictable. It would be unwise to pretend that we know how long these would take to fix. In such situations either use the default estimation, or don’t estimate at all.
A bad estimation is in many cases better than no estimation, especially since over a long period of time the average of the estimations can be rather close to the average time it actually took to fix a bug. Don’t fear being wrong, just make sure that everybody in the team understands that estimation is not a deadline. It’s applicable to features, but even more so to bugs.
It’s ok if you sometimes overestimate and sometimes underestimate, but you should adjust something if you underestimate all the time.
There is unfortunately no silver bullet. What we described above just happened to be a good setup for us, but each team will need to find an approach that works best in their particular situation. It doesn’t have to be one of the solutions described above though. You should never stick to all of the rules of a given framework — it’s always better to find a combination of techniques that works best for you, or even come up with something entirely different.