DEV Community: Przemysław Paczoski

Separate or project repository to store tests? Which, why, and when?

Przemysław Paczoski — Tue, 14 Dec 2021 09:39:42 +0000

Oryginally published at: https://kodziak.com/blog/separate-or-project-repository-to-store-tests-which-why-and-when

While starting to work with end-to-end tests, one of the first questions is where to store them. Alongside source code or in separate repository? Two solutions, both slightly different. Each of the approaches has their pros and cons – but which to choose?

In this short article, I've tried to describe them both. They're different, and from your perspective you've to choose which one to use, which isn't simple at all. I believe that this could help you a bit.

Project repository

Store tests alongside source code keeps everything in the same place. You've both app-related and tests-related stuff like tasks, workflows etc. in the same spot. It causes, that the developers could execute everything much easier than searching some commands in another repository. It closes the gap between the developers and test engineers (SDET's). There aren't any barriers.

When the tests fail, it would fail on the programmer branch, so it'd be his job to maintain them. That's the mental shift when things are together. You wouldn't be responsible for bringing back the tests to love after every failure. Or find the person which going to fix them or the development is blocked.

The point is, that storing everything in the same repository, there aren't any other “strange” place where some code checks the developer's job and makes it easier to maintain.

What's more, when the newest feature change the application flow and breaks tests, you don't have problem with different versions of tests and app. They're synced because of the same repository. No concern with merges and only one thing which left would be to maintain them on the same branch as the feature is and push a complete working solution. The time when tests aren't compatible with the newest application version shouldn't exist.

Example structure

<application-repository>/<root>/app

<application-repository>/<root>/tests

Separate repository

When the test code is in “another” project, and it usually breaks — the developers probably won't care about it. It's going to be your job to maintain the tests and unstuck the blocker.

Unfortunately, when the things are separated, it would reduce the collaboration between team members. If your organization have clear distinction between the developers and test engineers, a separate repository will widen that gap. One wouldn't work on each other solutions. That's really not a situation you should go. The most beneficial would be working together on both, the application code and tests, as a team.

Imagine a situation. From the developer's perspective. You've created a feature which breaks some of them and want's to merge it, fast. Obvious is, the tests would fail, so you've to set up to another repository, maintain the tests and prepare request, or just merge the feature without properly working tests. It would create a gap because it's impossible to merge both, new versions of the code at the same time.

In that scenario, each of the repositories going to have workflow according to its needs – which is good. It will separate the contexts, so the person which going to start working on one of the projects will have only things related to that. The application has tasks related to build and running the application, and in tests will contain jobs to set up the application under tests and execute them. But it has downsides too. It's just annoying to run the whole flow on local machines. You'll need to execute commands in different places, which bring up the complexity.

Of course, I didn't say that only the test's repository has to contain tests – nah. The application should have their own. The developers should write unit tests alongside the source code and execute them in pipelines.

Separating the repositories keeps more clean environment. Repositories contain only things it's concerned about. It's make easier to navigate on the daily basis. But there's a downside to this approach. You'll have more moving parts to your workflow, which makes things more complicated along the way. For example, with new developers, setup everything it's going to be a little more difficult in front of a situation, where everything is in one place.

Example structure

<application-repository>/

<tests-repository>/

Conclusion

Project repository

Increase ownership through the team,
Closes the gap between the developers and test engineers,
No problem with merging, you'll always have synced versions between tests and application,
Everything app and tests related going to be in the same place.

Separate repository

Decrease ownership through the team,
Widen the gap between the developers and test engineers,
Could be a situation where tests aren't working on newest application version,
Cleaner environment, only things concerned about are in the repository.

Reading the article and points above, it seems like a single repository is a straight winner. But it's just how it works for me, most of the time. It doesn't mean, that it's the best solution for your organization. It always depends – on your project architecture, company structure, how many people you have and how you're working as a test engineers.

The best would be to talk to your teammates and decide as a team which could work better for you. Work together on that topic. I assumed that you have in-house test engineers. If not, then, probably you should choose a separate repository for the tests.

Six things you should consider while designing a test architecture

Przemysław Paczoski — Tue, 23 Feb 2021 14:20:55 +0000

The beginning of automated tests in a project is easy and difficult at the same time. You can start smoothly, using base architecture, simple tests, etc - and it's gonna work! But... With growing architecture, solutions you'll take could be very hard to develop in the future.

That's why, you should consider a few things, before the start. I want to show you the problems and solutions do they solve.

Page Object Model

POM - Page Object Model is a design pattern that is commonly used in test automation for avoiding code duplication and improve test maintenance.

This model contains classes that represent web pages. You break down each page into a corresponded page class. This class contains locators (selectors) to web page elements and methods that perform actions on them.

This fills the most common actions you'll be doing in test automation - find the element, fill the value, click the button. For small scripts, you can store selectors in the tests, but with growing suites, you'll face many duplications - that's bad, you don't want them. If one of the used selectors gonna change, you'll need to update every place, you've used it.

And... POM is the solution. You'll create a separate class that all of the tests gonna use. Then, if any of the selectors change, you'll only have to update one place. It's faster for the person which updates it.

But... There is another model of creating an architecture for automation tests. It's called App Actions. I've never tested it, so I can't recommend it, but here's a nice article about it - click!

Logging

You should use a logging system to give the user (developer, or tester) enough information about what's happened in the tests while executing in a relatively short time. It's most possible, that you'll run your tests through a remote machine. If they fail, you need to know fast what's happened. Troubleshoot or reproduce failed tests with a stack trace and the latest assertion message only will be very hard.

That's why you should design a logging system.

A well-designed strategy will show the user step by step, what's happened in the test. In the project that I've worked on, we created a three-step strategy:

begin,
info,
end,

To do this quickly, we've used decorators (TypeScript only).

For example (code base on puppeteer):

@logger
function getText(selector) {
    return page.$eval(selector, (el) => el.textContent);
}

Result with:

[BEGIN] - getText from: #email
[INFO] - getText result: sampleemail@gmail.com
[END] - getText from: #email

The assumption was simple. We want to know when the method starts, what's the result and when ends. We've used a method name and parameters as a recognizer. That comes out with well-structured logs, which helps us to know exactly what happened in test execution.

Also, you can use these logs in your reporting system, which is described below.

Another good idea to mention is to write custom, detailed assertion messages. For example,

Expect: 5, Received: 8

It's saying not much. Without the context and knowledge of what the test is checking, it's hard to assume what's happened. You'll need to check it manually. But...

We expect 5 elements on the listing, got 8.

It's easy to read and clearly saying what's happened. From the message, you can easily assume where's the problem.

Also, there are two more areas, where detailed messages could be helpful:

timeout exceptions - ex. internet connections problem,
architecture exceptions - ex. DB connection, auth connection problem.

I believe, that having a proper logging system could make your work easier and speed up the troubleshooting process.

Test Data

Most common action in your tests gonna be to fill the value within the data. But... You need to have, or generate it. It's one of the most time-consuming actions, but necessary.

Let's split the data into two types:

static data - the one you store in database/repository/somewhere,
dynamic data - the one you generate using tools like a faker, etc.

Using class patterns you can design a system that handles the data for you. Connections to a database or dynamically creating sensitive data (ex. from faker.js).

Parallel Executions

Parallelization is an automated process that automatically launches your actions simultaneously. The goal is to save execution time by distributing tests across available resources.

Consider a situation. You have a big database. Execution time takes 50 minutes to complete. You have 50 tests, where one approximately takes 1 minute. Tests are isolated (if not, you need to work on them). Using 5 parallel executions, you can reduce the total testing time to 10 minutes.

Fast, reliable tests are key to a stable product.

One, the interesting out-of-the-box solution is Knapsack Pro. It helps run your tests in a parallel efficient way.

Deployment Pipeline

A deployment pipeline is a process, where you take the code from version control and make it available for the users in an automated way. The team, which works on a project, needs an efficient way to build, test, and deploy their work to the production environment. In old fashion way, this was a manual process, where there were few deployments per week, now it's a few per day/hour. A set of automated tools doing it for us.

Imagine a problem. You have 50 tests and execution takes 50 minutes (without parallel executions). To test the fresh version of the code, the developers need to build an application locally and run the tests against it. And okay, the tests found an issue, so the user fixes the code and does it again. Hours and nervousness growing. I'm pretty sure, that after a few rounds they're not gonna use the tests. You don't want this.

The solution is to use an automated deployment pipeline. It's designed especially for cases like this.

Combining it with parallel executions, you will create an efficient, fast strategy, which gonna defend your production with the tests. Even better, if failed tests block the deployment. You'll never push broken code to the real users. 🥳

Reports

Reporting is about documenting the process of test execution. They should combine a summary for the management and stakeholders. Also, should be detailed to give the developers feedback. Especially, when something fails.

Consider a failed execution. Tests are running on a remote machine, and somehow, you need to know what exactly happened. Why they failed and what's broken. Access to plain logs could be painful in searching this one failed test. A well-created report should cover that.

Each report could be different, but in my opinion, there are a few must-have metrics:

machine/environment name,
total number of running tests,
duration,
list of all test cases with logs (steps from logging system) in toggle,
test result (for every test case).

Have I mentioned failed test executions and it's crucial to know what happened? 🥳 Constantly looking at a deployment pipeline dashboard or searching in logs isn't that efficient. A well-written report, mixed with the slack notification can speed up this process much.

Conclusion

I've described six topics, which are really nice to consider in every mature test automation project. They solve problems, which you'll encounter on daily basis. By doing some coding and research, you'll easier your work and make the architecture expandable. Using a few hacks, you'll speed up your troubleshooting process and with automatically generated reports - you'll be as transparent as possible with your tests.

I believe, that this knowledge will help you at the beginning of your journey.

What's also good to consider:

base architecture - pattern to re-use base methods, create browsers,
video recordings - record whole test session,

Have a nice day! 🥳

Importance of CI/CD and parallelisation in test automation

Przemysław Paczoski — Mon, 01 Feb 2021 12:27:47 +0000

Oryginally posted at: https://kodziak.com/blog/importance-of-ci-cd-and-parallelisation-in-test-automation

Nowadays, products change very often. We're create new versions of applications on daily basis. That means we need to know fast if everything works. To achieve this, we could run tests manually before every release or...?

Let's automate this process.

How can CI/CD help with test automation?

Let's be honest. If we tell the developers "you need to run the tests manually", they probably won't do it too often. Why? Because they are time-consuming. The execution usually takes more than an hour. Maybe it will work a few times, but I'm pretty sure there will always be some kind of excuse. But that's life.

Then we're coming to the problem that we don't want to merge untested code, ever. We want to run unit, integration, and end-to-end tests (of course, if we have all of them:)). In a convenient way of course. Without any extra work from the user.

The worst we can do, is to make something dependant on us. Continuous processes require knowledge from the developers how to maintain the tests and that's very close to writing them. When we all work on one product, we're all responsible for its quality. And to achieve that, we should work closely, together. Who said that only testers could write E2E tests? Anyone from the development team can do it.

Last but not least, in continuous processes we can block the deployment when a test suite fails. Then the author needs to fix the feature or maintain the tests, if changes affect them. Of course, deterministic tests are needed here, which we all know - is hard to reach.

Loooong time execution...

Yeah... The problem we all know. Many times I've heard that tests takes one or two hours to finish.

We should avoid that.

Try to stick with 10-20 minutes to execute the core ones. To reach that, we can use parallelisation.

We can distinguish two types:

run tests by groups,
run multiple tests at a time.

Tests by groups

First, we should split our tests by groups/criticality. Recently I've written an article about that: What is smoke, sanity, regression testing? How to CI/CD with them. I've split the tests by types. And with that, we could run the groups sequentially.

For example, yours pipeline could looks like this:

An approach like that, makes sure that the most important user flows are always gonna work. Then, on production environment we could check these less important ones. That's fine, until we're aware that something could be broken.

You'll secure our time-to-market, while still checking everything you're supposed to.

Multiple tests at a time

Secondly, use parallel nodes. Basically, most of the tools are configured to run one test and then another. With a strong machine and properly configured tests you can run a few of them at a time.

One of the out of the box tools which solves this problem, and makes the execution faster is Knapsack Pro. A very good tool to get the best execution time you can imagine.

Spread knowledge about tests

But what if a test suite fails? What happens then?

There are two solutions:

you can always monitor what happened and try to find a solution,
you can make it easier for people to get the information about what happened,

And I really love the second option.

I think everyone who works on software should be able to write, maintain, and fix the tests. You're working together to deliver the best product quality you can. That means you should avoid the stereotypical division between testers and developers. You're working together, as a development team.

You can also implement automatically generated reports. Showing people, business, and other stakeholders what the tests cover, builds trust. And the trust in tests is what you need.

Conclusion

You should deliver the best product quality you can in reasonable time. It's good when you can do it fast and with fine user experience (our internal users). People won't run tests manually, so you should automate it. That's why you could use continuous processes.

Combining it with parallelisation, you'll secure the time-to-market.

Spreading knowledge about tests makes more people feel responsible for them. If something fails - someone from the team will need to fix or maintain it. The more people work on tests, the better these tests are.

Who is a Quality Assurance Engineer and what should he know?

Przemysław Paczoski — Sun, 17 Jan 2021 16:34:14 +0000

Oryginally posted at: https://kodziak.com/blog/who-is-a-qa-engineer-and-what-should-he-know

Have you ever think about steps in the testing journey? Manual tester, QA, QA Engineer, Test Developer, SDET, TestOps? There are many ways you can develop in. But today, I want to stick with one - QA Engineer.

What does exactly this person do? What are the responsibilities and requirements? What should he know to work efficiently on daily basis?

I've summed up areas and described basing on daily tasks.

What does a QA Engineer do?

QA Engineer means Quality Assurance Engineer is a specialist whose activities are aimed to improve the software development process and product quality using engineering knowledge. Manual tests aren't optimal in most cases. QA engineers automate the most common cases, which saves us time and money.

Responsibilities:

write, design, and execute automated tests,
maximize test coverage,
determine the priority for test scenarios,
create execution plans for these scenarios,
write documentation for automated processes,
build automation frameworks,
set up continuous integration

And to reach the QA Engineer job, you need to know various things. I've categorized and described them.

Methodologies

You'll be obligated to write, design, and execute automated tests. It means you have to know how to do it. Writing automation tests isn't something very different than you're doing manually. I mean, to write this one, you have to perform these steps manually. So you have to take exactly the same steps as a user, and write this as a script. Experience with manual testing will let you write these scripts better.

From time to time, you'll also decide what's important, and what's not. What should be tested next, and how to maximize test coverage. Experience and knowledge about methodologies will help you with these decisions.

The priority of scenarios? Another important topic. Creating complex test solutions, most sure you'll face a problem with them. What should we automate first and why? What should we run first? These are hard questions, but important. We should always focus on more important parts of the application. Because we don't want to show the user some unexpected behaviors, bugs, or unavailability of the website. They'll lose trust in our product, which we can't afford.

Programming language

Writing tests, creating test frameworks and architecture is the same as with applications. To do this, you need to know a programming language.

Some of the wanna-be test developers, decide to learn frameworks first. Without the proper programming knowledge, they're trying to write tests. Probably this gonna work, but this solution won't be maintainable so much. While growing, they will encourage more and more problems, because basics were created wrong. Fix these mistakes lately, will cost much more time and afford than proper planning.

But language knowledge isn't in my opinion enough. You should also be familiar with the concepts of programming. KISS, YAGNI, DRY, SOLID - right now, they could sound for you like a buzz word, but believe me - you'll use them daily. Knowing them will give you the opportunity of creating more maintainable and better solutions.

Sometimes you'll also need to pass framework methods. Instead of using a page.click() you'll need to perform language native one. Sometimes, you'll need an API call that your test framework doesn't wrap up. You'll face many cases, where knowledge of the language is important.

Languages:

JavaScript/TypeScript,
Java,
C#,
Python

Test tools

To write a test we're using test frameworks. These are tools set, out of the box, which gives you an easy way to control the browser. It means, writes a user like a scenario end's with a few methods. We don't have to reinvent a wheel, write from scratch integration with browser, etc.

Tooling is important, but concepts are more. While you know one test framework, you can easily switch to another. Most of them are very similar, it's like with a programming language, while you know the concepts and knows how to program, a language is literally a tool.

Example test frameworks:

Playwright,
Cypress,
Selenium

Processes

One of the bigger challenges is to set up pipelines. Nowadays, we're using Continuous Integration practice. Design and build a whole process is a very valuable skill.

You need to understand how deployment works, where you can run tests, how to do this?

To reach this, be familiar with topics like:

Docker,
CI/CD - Gitlab, Jenkins,
Cloud - AWS, Azure

There's a role called TestOps which is a connection between operations and testing, but I think mature QA Engineers should know the basics about processes, to be able to set up them from scratch.

Conclusion

QA Engineer role requires knowledge of many different areas. All of them affect the final solution, which with this knowledge will be better.

Knowledge of these areas also allows you to develop in various roles. Dive into processes? You can go into the TestOps position. Development? Then pick SDET.

Have a nice day! 🥳

What is screenshot, UI, API, and performance testing?

Przemysław Paczoski — Mon, 28 Dec 2020 15:32:40 +0000

Oryginally posted at: https://kodziak.com/blog/what-is-screenshot-ui-api-performance-testing

Building a product, we face the moment that we need more types of tests. Screenshot, UI, API, or Performance? There are many of them and every one is designed for other purposes.

Every type is different, and can help us with other things. I think, most of them should occur in mature projects, but for sure - you should know all of them.

So... What is...?

Screenshot testing

What is screenshot testing?

Screenshot testing is a technique used by end-to-end testing which captures a screenshot from a given URL and then compares the result with an expected image.

The most common usage of this type of test is to check if the user interface (UI) isn't broken and the page has been rendered properly.

Why you should do them?

Coming from old internet explorer to newest version of chrome. Using mobile and desktop in various OS. It's hard to check everywhere if our application renders properly. Not to mention a problem, where the developer will break something accidentally. A simple change could destroy our design.

How you can do them?

You can capture bulk screenshots through different desktop and mobile devices running in the various OS. All of them, you can compare with an expected image (snapshot). If the test will pass, you can deploy a new version of the application, otherwise, with the designed flow, you can ask a manager or designer to accept or reject changes. A simple diagram, I've prepared below.

Deployment diagram

But there is one thing you need to remember. You need to work closely with designers and developers. It's their job to accept changes in UI or revert them to the developer if something's broken.

Popular tools

UI testing

What is UI testing?

UI testing is a technique used by end-to-end testing whose main aspect is to check how the application handles user actions performed via input devices - mouse, keyboard, and others. Another one is to check if displayed elements on the page are interacting correctly.

Why you should do them?

Imagine that by an accident, your application has a faulty UI element, which is core for your business. Your users won't be able to get what they want, book an appointment, or buy a book. You will lose money and for sure clients, because with no trust in the application - you can't build the value.

By that, UI testing is fundamental. Tests will behave like a real user. Will interact with your application and check the output. What's better, they will do more than check the presentation layer. A complex test needs the business and data layer also working.

How you can do them?

There are various tools created for control browsers. Some of them, you will find at the end of this caption. These tools will give you the possibility to create a test, which behaviors should be designed as a real user. Running them every every deploy, you'll always know, if somethings break. Also, you can set up these tests in the deployment pipeline.

But... Not all that's pretty. These tests are slow. By slow, I mean that everyone needs to load a page, interact with it, probably reload, etc. This takes time. Many tests, takes much time to create an output, so it's important to run only the core one. More about it, you'll find in my latest article - click!

For sure, you shouldn't give up on them. We always need to provide a complete working application.

Deployment diagram

Popular tools

API testing

What is API testing?

The API testing is a set of actions, which includes sending calls to API to get output and validate the response against defined input parameters. They determine if the API returns the correct response or react correctly to failures or unexpected behavior. Instead of using standard user inputs, we use the software.

They're different than UI testing because don't concentrate on the presentation layer and the feel of the application. The main point is to check the business layer.

Why you should do them?

When you modify code in the backend, automated UI tests can be time-consuming and repetitive. With API testing we skip UI, which makes tests much faster. That's why we will know faster if our changes break business behavior.

How you can do them?

These tests can run similarly to UI ones. The main aspect is that you need a freshly build API, then you can run tests and base on the result deploy the new version of the application, or reject changes.

There are many various tools to perform API tests. Some of them, I've put below. Designing and writing API tests is crucial for business work. Especially, when your architecture is built on micro-services. You can check every one separately.

Deployment diagram

Popular tools

Performance testing

What is Performance testing?

Performance testing is a non-functional testing technique that determines the speed, scalability, stability, and responsiveness of an application under a particular workload. The goal of this type of test is to evaluate application output, speed, data transfer velocity, network bandwidth usage, and response times.

Why you should do them?

You need to know if application speed meets expectations on production. Performance is a key determinant of adoption, success, and productivity.

Running them, you will identify performance-related bottlenecks, which affect the user and created a poor experience - where you lost money. Bottlenecks are a single point or component within a system's overall function, that holds back overall performance.

You can also use them to compare two or more systems and identify the one that performs better.

How you can do them?

This is specified by organization and application. It depends on what you need and what the business considers most important.

You can use some of the tools, which I've listed below.

Deployment diagram

Researching this post I've found some interesting articles on how to build a pipeline with performance tests. I'm not sure if it's working well, but - why not to get a try?:)

Here a few links:

Popular tools

Conclusion

The listed types of tests has pros and cons. All of them are useful, and many you should have in your projects. Of course, everything depends on needs.

We've discussed:

Screenshot testing,
UI testing,
API testing,
Performance testing.

I hope, that this article clarifies to you a bit which types could be useful where and for what we're doing them.

Have a nice day! 🥳

How to make your UI automation tests resilient to changes?

Przemysław Paczoski — Mon, 21 Dec 2020 23:23:47 +0000

Oryginally posted at: https://kodziak.com/blog/how-to-make-your-ui-automation-tests-resilient-to-changes

Using CSS selectors to get elements in end-to-end tests is the most common way (besides xpath), but it can also be tricky and hard to maintain. One day everything could work fine and another a CSS class could be renamed. Styles could change without touching the business flow and then we could have a problem - we'll need to maintain the tests. Of course, the developers could do those changes for us, but they don't know our scenarios, our tests etc. It would be time-consuming for them, because they don't always know if their changes could affect our tests. And for these problems, I think I might have a solution.

But first let's talk about CSS a bit.

What are the common problems with CSS selectors?

Imagine that one day, you had to write a very ugly selector. The element you wanted to use was nested and the only solution available was to base it on CSS classes and HTML structure. And then, a developer made a little refactor that removed some useless HTML tags, which were of course, used in your selector. But he didn't change the business flow. And guess what will happen? Yes, you're right! The test will fail. Your selector will be broken and will need maintenance.

But let's take a step back and think about what are CSS classes designed for? For sure, not to use them in end-to-end tests. Why? Because they're designed to style elements. Everyday there might be changes in the application that don't affect the business flow - which is the most important part in end-to-end tests - but are related with class names changes. That hurts when we need to maintain a test that didn't involve any user flow changes.

Two examples, one simple solution - find better a mechanism to get elements.

What do I recommend? Read below!

Use `data-test-id` for help!

For sure, they're controversial but there are two main advantages of using them.

First one is that they're bulletproof. What? I mean, when we use this attribute with an element, no matter if the developer changes the HTML structure or CSS classes, we still get what we want. Unless they remove the element or change the business flow, our test should work fine.

And second, as you may notice, the developer will see that attribute while working with the code. It could help because they can notify us about possible changes or just maintain the tests if needed - using data-test-id makes it easier find occurrences in tests. And if we have more people working on end-to-end tests we can build more reliable solutions. That's what we all want to achieve - always working product:)

More about working closely with developers below:)

But should I use them everywhere?

Nope. Only with the elements used in tests. We don't want a mess in the code where almost every element has this attribute and only a few of them is used in tests. We want to build trust towards the tests and somehow show the devs that their changes could break our work.

Of course, we'll add some code, but with babel or similar solution we'll get rid of data-test-id on production version. Also, you can take a look at facebook where these attributes are used;).

A little bit of extra work can bring us many benefits.

What are the benefits of using them?

For sure, we can build communication between us and developers. How? Talk. If they see this kind of attribute with an element they want to refactor, they have to maintain the test (in the best case) or just notify us that something could affect it. It's beneficial to know when something can break our tests before running them on testing environment. It'll build trust to the tests. They won't fail because of changes which don't affect the business flow. It will help all of us.

The only thing we need to do is give the developers an opportunity and reason to maintain the tests.

Conclusion

Code structure can always change and it shouldn't affect our tests unless the business flow changes. With selectors based on CSS it could. That's why we decided to go with data-test-id which seems like a reliable solution.

They have some shortcomings but overall I think there is more pros. If you use them you'll work closely with devs which could result in other interesting solutions:) Just remember to add them wisely - only in the elements you're using in tests:)

Have a nice day! 🥳

What is smoke, sanity, regression testing? How to CI/CD with them.

Przemysław Paczoski — Sat, 12 Dec 2020 19:14:55 +0000

Oryginally posted at: https://kodziak.com/blog/what-is-smoke-sanity-regression-testing-how-to-ci-cd-with-them/

With growing test architecture, we are facing various problems. Execution time, flakiness, duplicate test cases etc. Tests are becoming hard to manage and we may start to feel like writing and maintaining them is meaningless.

While lurking on reddit I came across a discussion about test types. Then it hit me - why can't we categorize them and instead of one long run, perform them group by group? We won't lose any coverage, and we'll reduce time to market. 🥳

Types of end-to-end tests

Sticking to E2E tests only, after reading the reddit discussion and thinking about the topic, I've decided classify them into:

smoke testing,
sanity testing,
regression testing.

You probably already know the last group very well (if not → click here).

Now let's dig into each of the types mentioned before.

Smoke testing

The main characteristics of these tests are:

They're performed to cover critical functionalities of the application (Priority1 scenarios),
The objective is to verify "stability" of the application to continue with further testing,
They're a subset of regression tests,
They're like a general health check up,
Failing these tests results in instant rejection.

Simply speaking, smoke testing should cover the most critical parts, the ones that let us confirm that the applications is working and delivers business value.

For example, this kind of scenario could be in this category:

[Netflix.com] User can log in, choose a tv series/movie and start watching.

Sanity testing

Next group is sanity tests, whose main aspects are:

They're performed to check more detailed functionalities of the application (Priority2 scenarios),
The objective is to verify "rationality" of the application to continue with further testing,
They're a subset of regression tests,
They're like a detailed health check-up,

In simple words, sanity tests should cover more specialized user actions.

Here's an example of a scenario that falls into this category:

[Netflix.com] User can search for movies using search bar.

As you can imagine, sanity tests may have some test cases in common with smoke tests but also there are certain differences between these categories.

Regression testing

Last but not least - regression tests:

They're performed to check detailed functionalities of the application (Priority1 + Priority2 + Priority3 scenarios)
The objective is to verify every existing feature of the application,
Regression test cases should be well documented,
They're like a comprehensive health check-up,

Simply speaking, regression tests should cover all the aspects of application - the important and less important ones. Executing regression tests should give us an overview that everything works properly.

For example, a scenario that could be in this category is:

[Netflix.com] User can open "more information" about a tv series/movie and choose genre.

Amount of tests by type

Regression tests > Sanity tests > Smoke tests.

And here's a visual representation of how the sets intersect.

Arrange a deployment process based on test types

I've worked with one test database and deployment process for a while, and I figured that it's not perfect and not bad, but as it grows, it needs some changes.

Depending on the type of tests, we can choose the most important ones and those that are less important. As we all know, the most important ones are those that deliver the business value, so we have to check if those functionalities work as expected with every deploy. Based on these criteria we can design our deployment process.

Working with an application based on CI/CD carries potential challenges such as designing a deployment process with automation testing. Because we don't want it to be long, we can categorize the tests, and run each group at a different time.

As you can see, only smoke tests are run with each application deploy. This way we'll always know that the most important functionalities work.

Conclusion

As the number of your tests grows, they become harder to maintain. Running all of them after every deployment consumes too much time. If you organize them into categories you will reduce time to market without losing any necessary coverage.

How you categorize your tests depends on you. They don't necessarily need to be divided into smoke, sanity and regression categories. You can do it differently, for example, by criticality level.

Automated testing in deployment process is a standard practice in modern software development, used by the best teams and companies. CI/CD should therefore depend on the results of our tests, which will ensure that the critical functionalities always work.

Have a nice day! 🥳

First steps with end-to-end testing using jest + playwright (puppeteer)

Przemysław Paczoski — Thu, 03 Dec 2020 19:06:46 +0000

Oryginally posted at: https://kodziak.com/blog/first-steps-with-e2e-testing-using-jest-playwright

This post is a continuation of the “First steps with test automation” article.

Recently I wrote a post about my first steps with test automation. I tried to answer questions related to the topic and guide you where and what to search for (if you're interested - click here!). Now, I want to expand on it with technical steps you need to take and show you how easily you can create a basic test with a real world example (my blog 🥳).

We will create a base repository with Playwright and JestJS and write a basic test which will:

Load the blog page,
Check if the blog title contains the author’s name,
Check if the posts list isn't empty,
Click on the first post title and check if it loaded correctly.

To achieve it, I think you need a little more knowledge, so I split the article into three topics (+bonus):

What do I need to start with automation?
Project setup with Playwright and JestJS
Let's write an automated test
Bonus: Test recorder!

Before starting the game, here are some topics which you should be familiar with:

Basic knowledge of JavaScript,
You know how to use CLI and NPM/Yarn,
You're familiar with HTML/CSS.

What do I need to start with automatization?

First of all, you need a library/tool which allows you to control a browser using code. Today, I'll use NodeJS library called Playwright (which is very similar to Puppeteer – here you can find a nice article about the differences: https://blog.logrocket.com/playwright-vs-puppeteer/) to automate our actions with an API. You need a tool like this to write tests that will imitate the behaviour of end user (behave like our clients).

Basically, doing a test manually, you're using a mouse to click and scroll or a keyboard to fill in the forms. In automated tests, you're using a library API to do the same things but the browser is controlled by automation software, not a user.

Next, we need assertions. Playwright provides their own ones, but honestly – I've never used them. I like JestJS, which, besides that, is a test runner, which gives you more opportunities to control the whole test base. With an easy way to do parallelisation or configs for dev/production envs, writing tests is only a pleasure 🥰.

In summary, you need:

A library to control a browser → we'll use Playwright,
A test runner → we'll use JestJS

Project setup with Playwright and JestJS

So we will use:

And if you decided to stick with puppeteer:

Note: When using puppeteer, you need to change selectors because the API is slighty different. In our basic test, it is enough to remove css= part and change waitForLoadState ⇒ waitForNavigation. After those two changes, everything should work.

Before we start writing our first automated test, we need to set up the repository:

Let's initialize it first:

yarn init or npm install
git init

Then install the required dependencies:

yarn install jest jest-playwright-preset playwright

npm install jest jest-playwright-preset playwright

And create two config files,

// jest.config.js
module.exports = {
  preset: 'jest-playwright-preset',
}

// jest-playwright.config.js
module.exports = {
    // Describe which browsers we want to run
  browsers: ['chromium', 'firefox', 'webkit'],
  launchOptions: {
        // If we want to run browsers in headless mode or not,
    headless: false,
        // If we want to have opened devtools from start
    devtools: false,
  }
}

Also, we need to an execute script to our package.json:

"scripts": {
    "test": "jest"
  },

Now, you can yarn test or npm run test which will run all files matching *.test.js names – but we don't have any tests yet 🙈

One more thing that I added is tests/ directory where we will store our test.

Let's write an automated test

First we need to know what we want to test 🙈

I created a basic flow relating to my blog, which you're reading now:

Check if the blog title contains the author’s name,
Check if the posts list isn't empty,
Click on the first post title and check if it loaded correctly.

In the directory tests/ I added a file example.test.js. And to make it easier for you, I coded the above scenario, adding comments line by line, describing what’s happening.

let blogPosts = [];

describe('Blog - kodziak.com/blog/', () => {
    // Will trigger methods before tests
  beforeAll(async () => {
    // Load blog page
    await page.goto('https://kodziak.com/blog/');
  })

  test('title should be "Blog | Przemysław Paczoski"', async () => {
    // Get website title
    const title = await page.title();

    // Compare title of current page
    expect(title).toBe('Blog | Przemysław Paczoski');
  })

  test('should display list of blog posts', async () => {
    // Get all blog posts as an array of objects
    blogPosts = await page.$$eval('css=.post', elems => elems.map(el => {
      return {
        title: el.querySelector('.post-title').textContent.trim(),
        description: el.querySelector('.post-description').textContent.trim(),
        href: el.href,
      }
    }));

    // Check if list length is greater than 0
    expect(blogPosts.length).toBeGreaterThan(0);
  })

  test('click on blog post should redirect to article', async () => {
        // Go to first blog post, there we're waiting to resolve all promises from array
    await Promise.all([
      // Click oo .post-title css class element
      page.click('css=.post-title'),
      // Wait for networkidle event, promise resolves after event
      page.waitForLoadState('networkidle')
    ]);

    // Get title, content and href of current article
    const [articleTitle, articleContent, articleHref] = await Promise.all([
      // Get article title, we use here array destructuring
      page.$eval('css=h2', el => el.textContent.trim()),
      // Get article content
      page.$eval('css=.content', el => el.textContent),
      // Get article href
      page.url()
    ]);

    // Destructuring first element from an blog posts array. First we use array destructuring and then object destructuring
    const [{title, description, href}] = blogPosts;

    // Compare title from posts list with current article's title
    expect(title).toBe(articleTitle);
    // Check if the current article content includes description from posts list, compare output with boolean true
    expect(articleContent.includes(description)).toBe(true);
    // Compare href from posts list with current article's href
    expect(href).toBe(articleHref);
  })
})

After we finish coding our test, let's run it using the command yarn test.

Playwright will open three browsers which we described earlier: chromium, firefox, webkit.

And if everything goes well, we should see the following log:

And that's it! We ran our tests positively in three different browsers.

Bonus: Test recorder!

Lastly, a good way to start with test automation could be to use test recorders. Nice one, created with playwright is: https://github.com/microsoft/playwright-cli which allows you to record your browser interactions and automatically generate a fully working code – an automated test with Playwright.

Conclusion

In this article, I wanted to show you how easy it is to automate a basic test scenario using tools such as Playwright and JestJS. I hope that will be a good start for you to explore this part of software engineering.

After reading it, you should have learned:

What Playwright (Puppeteer) and JestJS are used for,
How to set up a repository using those technologies,
How to write a basic test.

Fully working, implemented code from this article can be found in my repository: https://github.com/Kodziak/playwright-first-steps

Have a nice day!

First steps with functional test automation

Przemysław Paczoski — Tue, 01 Dec 2020 23:29:10 +0000

Originally posted at https://kodziak.com/blog/first-steps-with-test-automation

Starting your test automation journey isn't always as smooth as it should be. You may have a lot of questions like:

Do I have to be a manual tester before?
What language should I start with?
What framework/library should I pick?
What to automate first?
Where to search for information?

and probably many more. I'll try to answer all of them based on my experience and the steps I took when I started with automated functional tests.

So my case was:

experience with manual tests,
no programming background,
the team wanted to start with test automation,
the business was convinced that test automation would be useful.

And I’m going to write this article from that perspective.

Do I have to be a manual tester before?

The short answer is: it would be helpful.

I started my adventure as an intern manual tester and after quite a short time I moved to junior quality assurance position. The whole job was to perform and execute manual tests. I was doing it only for about half a year and then decided to go with automation. I knew that some regression cases could be easily automated and give me more time for other tests. And in my opinion it's a good place to start with automation. It will save you a lot of time for testing new features and probably shorten your time to market.

When you start with manual tests first, you’ll most certainly learn how to write a good test case which is a very important skill. I mean, doing it manually you’re checking specific elements and behaviours of the application so with this knowledge it would be much easier to write good automated tests. If you skip the part of being a manual tester, you will have to catch up – so another topic for beginners:)

That's why, in my opinion, it is important and could be very helpful to start as a manual tester, learn how to test, and then start with automation testing.

What programming language should I pick?

Like everything, it depends. When starting with test automation it’s good to have buddies in the team who could help you. It's possible if you're writing tests in the same language as they're using. I started like that and in my opinion this is the best way.

So, you're working in a scrum team, you're writing front-end applications and you have to write UI tests without a programming background. Having those prerequisites, I would go with a language that my teammates know. Especially in the beginning, they could help you to set up the architecture and write the first test. From a developer’s perspective, learning how another framework/library works is kinda easy. And problems we could face may vary – a problem with framework and a variable not being passed, bad syntax or wrong if statement. Having a bunch of people that could help you in those situations is priceless.

Another nice thing is that the whole team could write test scripts. We are all working to make the application as good as we can, so why couldn’t a developer update a test script after it was affected by his/her changes to UI. And having gone through a given test scenario, he/she could write a whole test. It would speed up our work and let us exchange knowledge. It's very beneficial when you're not the only person who can maintain the test base.

We all want to have a bug free product (and we all know it's impossible).

What framework/library?

Of course it depends on the programming language you have chosen. Here are the most popular solutions and combinations which I know (together with links to documentation):

Java ⇒ Selenium,
C# ⇒ Selenium,
JavaScript/TypeScript ⇒ Jest + Puppeteer, Cypress, TestCafe, Selenium, Nightwatch, Protractor
Python ⇒ Robot Framework, Selenium.

You can notice that Selenium gives you the opportunity to write in almost any language you want. And overall it's a nice solution with another big plus: most of the problems you’ll face with it have already been solved on StackOverflow, which is very helpful in the beginning.

What to automate first?

I've already mentioned manual testing and regression. As we all know, regression is kinda repeatable. Many times, we're doing the same cases all over again which is quite boring. Also, it requires quite some time: sometimes a few hours, sometimes days. So if we write all the cases as automated we could save a lot of time for feature tests and reduce time to market.

Regression test cases aren't very dynamic either. I mean, we probably won't be changing half of the tests by deployment. They don't need much maintenance time.

Reduced time to market is also an important thing related to automated regression testing. Doing it manually could take, for example, two days, and when we switch to automation, we could make it, for example, one hour. It's very beneficial for business, which could be an argument for starting the automation. And one hour is still a pretty long time, in my opinion. Using parallelization or other techniques we could shorten it down to a few minutes. But it depends – of course.

So I would start with regression and probably with login test cases. It’s repeatable, well-known and not so hard.

Where to search for information?

I would stick to the official documentation, which is linked above, but you can also use:

There are two aggregates of links and general websites which could be helpful. Honestly, most of the time I'm using the first two ones because typing any problem into Google will probably return a solution from StackOverflow.

Summary

I hope I answered the given questions, but let's sum it up:

Do I have to be manual tester before? → in my opinion, it’s very helpful but not necessary,
What language should I start with? → pick the language that your team knows,
What framework/library should I pick? → related to the chosen language, the list above,
What to automate first? → regression tests and login test case are kinda nice to start with,
Where to search for information? → documentation + Google and StackOverflow for problems.

It's just my opinion based on my own experience. If I had to start with automation again, I'd do the same, with those prerequisites. But of course – you have to adjust these steps to your business, type of work, and the situation you’re in.

Have a nice day!

Why do most test engineers fail interviews?

Przemysław Paczoski — Sat, 21 Nov 2020 07:36:52 +0000

Originally posted at https://kodziak.com/blog/why-do-most-test-engineers-fail-interviews

Recently I had the pleasure of interviewing candidates for my QA team. Sometimes, they were hard. You may ask why so I'll try to explain. I hope that in the future we can teach and prepare wanna-be test engineers to pass every interview they want to:)

Our roadmap for this article:

Basics of the language
Compatibility with the team
Problem-solving
Conclusion

Basics of the language

In my opinion, knowing the basics of the language you want to use to write automated tests is quite important to write them well.

Many wanna-be test engineers (and not only beginners) know how to write a test, but when it comes to creating a custom solution - it could be hard without the basics of the language. In my daily work I'm facing various problems. Sometimes I'm just writing tests, which is kinda repetitive and easy, but there’s been many situations when we had to create some custom solution to simplify our work (like our custom runner, based on jest or TestKeeper which is an application to handle tests by micro-services). In this situation, I'm pretty sure that they could write this, but the quality of the code would probably be awful and hard to maintain. It's not the point:)

And to give you an example. One of my questions was very simple: what’s the difference between let, const, and var? It's a very basic question often asked in interviews. And you know what? Many of the candidates couldn't answer it. I was kinda shocked because it's like the first question from the Top Ten JS interview questions.

Compatibility with the team

Another important aspect of the recruitment process is your compatibility with the team. I mean, if your approach and personality doesn't fit the team – you won't be processed to the next stages. And there’s nothing wrong with that. It's not your fault but you have to be aware of this.

I'm mentioning this because for us, leaders, it's a very important topic when building the right team.

But... I can give you some tips, what skills a good tester should have:

proactivity,
problem-solving,
business point of view,
flexibility,
quick learner,
collaboration and social skills,
communication skills,

Problem-solving

As you may have guessed, the quality that I particularly value is the problem-solving attitude. And it’s essential not only during the interview but also on a daily basis. But what does it mean?

In my opinion, a good tester shouldn't be afraid of confronting problems. Sometimes people come to us and ask difficult questions – it's just daily work. We should try to face them and if at that moment we don't know the answer – ask around and try to find it. Probably this asking part will be connected with doing something so treat it as a challenge.

In my recent interviews I met various candidates. Some of them, when given a task, tried to solve it, even if they haven’t encountered a similar problem before. The answer from another group was "I don't know", and it was wrong. I mean, when you don't have experience in a given topic, just say it and try to answer what you feel it could be. I'm not saying it's the best approach every time to every question but I really like it when someone tries to give an answer with their own words, just thinking about how it could work. We could at that moment recognize how each of the candidates handled challenges and problems.

Conclusion

Going through interviews is hard and most of the time – stressful. It's not a piece of cake but proper preparation before the talk should help you get the job. In my opinion, as I've written above, you have to go with a proper attitude and thorough knowledge of the language basics. But of course it's not everything, these are just the fundamentals you should have. The rest depends on the position you’re applying for, your seniority and experience.

I hope this article gives you an idea of some basic problems which are easy to tackle and could boost your confidence in interviews.

Useful articles to help you build your future career:

Have a nice day!

Set up special Github readme!

Przemysław Paczoski — Sun, 20 Sep 2020 10:46:06 +0000

Originally posted at: https://kodziak.com/blog/set-up-special-github-readme

Some time ago I hit an article, that Github added a special type of repository, the ones you see above (we'll make it better), where we can write something about us. I think, that's a great idea to show, what exactly I'm doing, and maybe... it could give me some movement to my initiatives?

But there's one problem. I don't like writing about me, I just... don't. I can't do this right (but I'm writing a blog, huh).

The good one is that Github gave us a basic template, where we just can edit some entries and voila, we'll have something similar as I have.

If you're curious how to add this to your Github profile - read the whole article.

How to create your own?

You need to create a repository, but it's a special type. It has to be named as your GitHub account. So just go to https://github.com/new, same as you'll have to create a normal repository, and when you'll type the name of your account (in my case it's kodziak), you'll be greeted with a message telling you about this special repository.

Highly recommend setting up this repository as a private to avoid showing a messy/broken page in time you'll be working on it.

Also, you can initialize base README.md file, which will automatically appear in your profile.

And voila - you have your own page.

Let's make it awesome

Doing research, I found some nice (nicer than mine, huh) showing pages. So I thought, why not make me more interesting?

I solicited my page into categories. As you may know, I'm also a podcaster (I hope, you know if not, shame on you - I'm kidding), so I'll go with categories:

Who am I?
My core actions (like blog, podcast)

And at the footer - some of my social media, where you can connect and write to me (feel free, I'm always answering!).

Assumptions I go with

I like emojis, so I'll go with like every header for a specific category that will contain one. I think it's the way we can add some colors and fun to this text.

But conversely to emojis, I like simplicity, so I thought without adding so many colors, images, gifs, etc, because it just not my style. All the time, I'm trying to keep it simple, to show this nice guy/woman who will come to my profile, as many pieces of information as he/she can get in just a few seconds. Not distracting him/her by jumping ducks.

Another rule I think you should stick is to make it short. No one wants to read long, boring texts (like this one). I'm a fan of short tips and tricks, wherein seconds I can get what I wanted - so I'm tried to keep my page in this style.

Conclusion

Make it fun, simple, and short. But make it. I could help you.

I hope, this article gave you knowledge on how to set up this and inspiration how can you improve (setup?) your page. If yes - feel free to write to me with a link, I'll put it there!

Feel free to write to me with any topic 😇

Have a nice day!

In case you need inspiration, there are some interesting ones:

DEV Community: Przemysław Paczoski

Separate or project repository to store tests? Which, why, and when?

Project repository

Example structure

Separate repository

Example structure

Conclusion

Project repository

Separate repository

Six things you should consider while designing a test architecture

Page Object Model

Logging

Test Data

Parallel Executions

Deployment Pipeline

Reports

Conclusion

Importance of CI/CD and parallelisation in test automation

How can CI/CD help with test automation?

Loooong time execution...

Tests by groups

Multiple tests at a time

Spread knowledge about tests

Conclusion

Who is a Quality Assurance Engineer and what should he know?

What does a QA Engineer do?

Methodologies

Programming language

Test tools

Processes

Conclusion

What is screenshot, UI, API, and performance testing?

Screenshot testing

What is screenshot testing?

Why you should do them?

How you can do them?

Deployment diagram

Popular tools

UI testing

What is UI testing?

Why you should do them?

How you can do them?

Deployment diagram

Popular tools

API testing

What is API testing?

Why you should do them?

How you can do them?

Deployment diagram

Popular tools

Performance testing

What is Performance testing?

Why you should do them?

How you can do them?

Deployment diagram

Popular tools

Conclusion

How to make your UI automation tests resilient to changes?

What are the common problems with CSS selectors?

Use data-test-id for help!

But should I use them everywhere?

What are the benefits of using them?

Conclusion

What is smoke, sanity, regression testing? How to CI/CD with them.

Types of end-to-end tests

Smoke testing

Sanity testing

Regression testing

Amount of tests by type

Arrange a deployment process based on test types

Conclusion

First steps with end-to-end testing using jest + playwright (puppeteer)

What do I need to start with automatization?

Project setup with Playwright and JestJS

Let's write an automated test

Bonus: Test recorder!

Conclusion

First steps with functional test automation

Do I have to be a manual tester before?

What programming language should I pick?

Use `data-test-id` for help!