DEV Community: Nicola Lindgren

Choosing what not to write automated tests for

Nicola Lindgren — Tue, 28 Dec 2021 13:09:02 +0000

This was originally posted on my personal blog: nicolalindgren.com

While it’s important to make sure your test automation has good coverage, I think it’s also important to know under which circumstances you should not write automated tests.

In my opinion, when it comes to test automation, more isn’t necessarily better.

More tests don’t necessarily result in better coverage.

More tests don’t necessarily increase your chance of finding regression bugs.

More tests is just that. More tests.

When deciding what not to write automated tests for, I think it’s important to ask yourself a few things.

1. What is your goal with your test automation?

Be specific and realistic about your expectations.

Come up with some specific examples of the types of regression issues that could come up and would be found from the test automation you have in place.

The best part of having a clear goal for your test automation is that it will guide you in knowing when and when not to write tests.

2. When deciding to add tests weigh the benefits of having the test against the costs

Is your test likely to find issues? Do you already have a test/ tests in place that would catch the issue(s) your possible test is designed to find?

3. Is this area of the app/website looking to change soon? If so, is it worthwhile writing tests for this area now or waiting a little bit?

A factor to consider here is how considerable the changes are looking to be and how far in the future changes for this area of the app/website will be.

Do you also write test automation? What makes you decide whether a test is or is not worth adding to your test automation suite?

If you want to read more interesting articles on testing, development, agile and automation, you can follow me on Twitter @nicolalindgren 👋

If you want to read more about when you shouldn’t write Test Automation along with tips on how to get started learning test automation, check out my upcoming eBook: Starting Your Software Testing Career – due out in January 2022.

Dealing with Intermittent Bugs

Nicola Lindgren — Wed, 22 Dec 2021 15:37:13 +0000

This was originally posted on my personal blog: nicolalindgren.com

Intermittent bugs can be a frustration for the entire team. You may be lucky enough to uncover it before it reaches any beta testers or you may find yourselves having to figure out what the hell happened because a customer (or several) has reported it but you are struggling to replicate it.

Here are some ideas on how to deal with intermittent bugs.

Gather information about the bug

This can be easier said than done as you might not always have easy access to the person who came across the bug. I’ve been on the receiving end of bugs with vague descriptions and no information on OS, device, error message etc. When possible, I do try to gather the following information:

When did it occur? (exactly what time)
Which device?
Which Operating system?
Were you able to replicate it later?
What error do you see? (exact wording)
Screenshots
Exactly what did you do before the bug

To be honest, I rarely receive all of this information, but I do ask for it in case I am able to get some of this information that can help narrow it down.

Look at the logs

One of the reasons I like to ask when it occurred (exactly what time) is so I can see if anything happened in the background etc. at the same time (or around the same time) that could have caused this to occur.

Looking at the logs can also give us more information as to exactly what happened; what the issue was.

Timebox the investigation

I find it can be easy to lose track of time and continue investigating but not get anywhere. I tend to timebox the investigation if I don’t receive any additional information.

I’ll let the product owner/delivery lead etc. know that I will look into this for X time, but also that I will park it after I have taken notes on what I have tried and what happened.

Which leads me to…

Record what you have tried (be specific)

Just because you haven’t replicated the issue, doesn’t mean what you have tried doesn’t have any value.

I like to be specific and write exactly what I tried (when, on what device etc.) and what happened, then add these comments against the bug in the bug tracking tool.

This can help you (or someone else) later when you need to investigate the issue further if you didn’t manage to solve the mystery the first time around.

Pair with someone

Pairing with someone means you can bounce ideas off another person. It can be easy to get stuck or hit a rut when you are trying to tackle something like this yourself.

If you pair with someone, they can give you ideas of things you haven’t tried yet and you can ask each other “what else can we try” or “what else could we be missing?”

Do you have other ideas on how you handle intermittent bugs? What do you do when you come across them?

Let me know on Twitter! @NicolaLindgren

How do you actually prevent bugs?

Nicola Lindgren — Wed, 15 Dec 2021 19:01:03 +0000

This blog post was originally posted on my personal blog: nicolalindgren.com

It’s not hard to find articles or pieces of research claiming that the sooner you find a bug, the cheaper it is to fix. But I’ve found there isn’t actually a whole lot of information out there on exactly how to prevent bugs in the early stages of a software development project. i.e. before code is written, while requirements/user stories are being written or just after the requirements/user stories have been written

Here I would like to share exactly how I help prevent bugs on projects and how I help others come up with ideas on how to prevent bugs as well.

During feature walkthroughs

Get yourself invited to these. They may be called something different at your company/project but they are essentially a meeting, presentation or discussion around the general gist of the feature you are about to work on. There may or may not be designs at this stage yet.

If designs exist, ask the designers for access to the designs before the meeting if possible so you have a bit of time to process your thoughts before the meeting, also ask if they are open to feedback before the meeting or if you should wait until during the meeting.

If designs don’t exist, I do find this trickier as I’m a visual person and designs help make things more “real” for me. There are, however, still ways you can cope with this.

Here are a few things you can ask about during feature walkthroughs:

Error scenarios – how will errors be handled; where will validation be done (backend or frontend) etc.
Ask for the backend documentation and how things will be validated there, then make sure that things have stricter validation in the front end (where possible), this helps prevent problems where the FE accepts data and you submit a form but the backend rejects it (e.g. mandatory fields, all mandatory fields the backend requires MUST be mandatory in the front end)
Are there any mandatory fields we need to consider?
Are there any other features we expect this feature to be consistent with? (For more on consistency heuristics, check out my blog post here) Some examples of this could include: how errors display, positioning and colours of buttons, user feedback when submitting a form (if you already have these sorts of things in place, you can then ask about these and if this new feature will be aligned with the existing behaviour within your site/app)
Are there any dependencies within the app we need to be aware of?
Are there any external dependencies (e.g. 3rd party ones( that we need to be aware of?
Along with the questions above, you can also help trigger further discussion by stating how you understand how the feature will be used in the feature walkthrough.

Yes, you may be wrong – but that’s fine. The value here is that by making your thought process and understanding explicit, then others can step in and correct you – this helps enable a shared understanding. You will also help others ask questions about things by verbalising your thought process.

Once the requirements/user stories are written

I’m going to assume here that the requirements/user stories are still subject/open to change, but limited to a certain extent. (no massive changes)

Make sure that your assumptions are made very clear – state them in your testing notes and also at the start of any testing discussion you have with your team

Have a testing focussed discussion with the developers in your team. Here you can share exactly how you will test and what you will expect – also make sure to highlight that you may have misunderstood things, and that you are open to testing ideas from the developers in your team. I tend to go with testing notes, instead of test cases – but ultimately it’s up to you to decide on what’s best for your project and realistically, what you can do a walkthrough with.

Where possible, use examples in your testing discussion, if there’s examples of test data you could utilise – then use these concrete examples of what you would be doing to test, and what you would expect in these examples.

For example:

Let’s say you are asked to test login functionality. Then you can write examples of email addresses and password combinations, and the error messages you expect to appear.

nicola@email.com / –> Please enter your password

nicolaemail.com / ********* –> Please enter a valid email address

After the discussion, make sure to share the notes you took and how your testing notes/ test cases have been updated to reflect what has been discussed.

Follow me on Twitter: @NicolaLindgren