DEV Community: Oleksandr Romanov

Books and courses about blockchain for (not so absolute) beginners

Oleksandr Romanov — Tue, 18 Jan 2022 10:46:21 +0000

Recently, several people reached me and ask - what resources I can recommend in order to build foundational knowledge about blockchain.

Here is the list of books and courses that I finished during 2021-2022 by myself.

Books:

Mastering Blockchain https://lnkd.in/ekpsMwKR Recommended.
Blockchain and decentralized systems https://lnkd.in/eMqHtzWk Highly recommended.
Grokking Bitcoin https://lnkd.in/eSNDneH7 Maybe, if you want to deep dive into Bitcoin.

Courses on Coursera:

Blockchain Specialization https://lnkd.in/eRFgwFsK Recommended for full beginners.
Blockchain Scalability and its Foundations in Distributed Systems https://lnkd.in/ejUSRs8Z Recommended for people with some background.
Decentralized Finance (DeFi): The Future of Finance Specialization https://lnkd.in/eEkRF8qU Recommended for those who interested in DeFi.

Courses on Udemy:

Blockchain and Bitcoin Fundamentals https://lnkd.in/e2k4Q_KD Nice and gently intro to the blockchain.
Blockchain Advanced Level: Uses Beyond Bitcoin https://lnkd.in/esDEQg2M More in-depth view to the blockchain world.
Blockchain A-Z™: Learn How To Build Your First Blockchain https://lnkd.in/eTcR6Ad2 Recommended.
Ethereum Blockchain Developer Bootcamp With Solidity (2022) https://lnkd.in/e-TEnNMh Recommended. Good way of teaching the code.

Blockchain testing - a mindmap

Oleksandr Romanov — Fri, 14 Jan 2022 07:45:03 +0000

In my previous blog post, I shared a useful information on what does it mean to "test a blockchain".

Today, I want to present a mind map of knowledge you need to learn in order to test blockchain and blockchain-based applications effectively.

Feel free to suggest things that I may miss in this map.

What does it mean 'to test a blockchain' and which knowledge do you need for it

Oleksandr Romanov — Mon, 06 Dec 2021 08:32:53 +0000

Photo by Akinori UEMURA on Unsplash

Blockchain is a hype topic nowadays. It seems that every other person is making their own cryptocurrency or opening a cryptocurrency exchange at the corner of the street :).

To better understand what blockchain is and how does it work, I highly recommend these introductory videos: 1 and 2

But what blockchain brings to the testing world? How deeply do you need to know about blockchain to test with such systems?

In this blog post, I will describe a few different cases of testing blockchains. Also, I will share my list of things you need to know to bring quality to such systems successfully.

Blockchain networks

It is the most "hardcore" option. In this case, the entire blockchain network is tested - as a holistic solution. For example - verification of Bitcoin, Ethereum networks.

What we test:

storing transactions in blocks and communication between them
cryptographic encryption and key management protocols
data replication between nodes in the network
consensus algorithms
system performance
system security
fault tolerance of the system

Blockchain-based applications

Here we are dealing with ordinary Web and Mobile applications, in which one of the functions is either to read something from the blockchain network or to write some data to it.

Communication with the blockchain on the backend side takes place through API requests. Often this is not an HTTP REST API but something more specific like gRPC.

The API can be either a third-party system or just a node of the blockchain network located inside the system.

What we test:

integration with blockchain
functional and non-functional application properties (as at any other application)
security and performance

Pay attention that the application's performance can depend on how fast blockchain works at the backend.

Cryptowallets

Wallets are applications that store information about your passwords and enable you to buy and sell cryptocurrency. They can be both in the form of web and mobile applications as well as desktop.

The main aspect when testing is the use of test blockchain networks (TESTNET) for transactions. The currency is technically the same as the real one in such networks, but it costs nothing.

In addition, there may also be specific types of wallets in individual electronic devices. And this is a separate exciting world of embedded devices.

Smart contracts

Smart contracts are software code (uploaded to the blockchain) that implements some business logic. Most often, smart contracts mean the code in the Solidity language in the Ethereum blockchain. But there are other programming languages available in different blockchains.

What we test:

unit tests for contracts
functional tests contracts
security testing

The unusual thing about testing is that it starts working immediately after the code gets to the blockchain (“deployed to production”). It is challenging to "stop" it. And it’s impossible to delete or revert the deployment.

Cryptocurrency exchanges

Exchanges are usually web (or mobile) applications that allow you to create an account and trade (convert) different cryptocurrencies.

They are very similar to ordinary sites for banking or currency exchanges.

The main difficulty of testing here is integration with APIs of many hundreds (or even thousands) of blockchains. So get ready for integration bugs and quick fixes.

What do you need to know and understand to test the blockchain?

Hashing: how does it work, and what are the different algorithms for it
Cryptography: symmetric and asymmetric, electronic signatures
Consensus protocols and their types
Distributed systems and methods of communication between a node in a network
Transaction models: UTxO (Bitcoin) or account-based (Ethereum)
What are the transactions, and how are they stored (hello Merkle trees :))
Types of blockchains (Mainnet vs. Testnet, open and permissioned)
Types of crypto wallets and principles of their work
Smart contracts (why are they needed)
Programming languages of smart contracts - Solidity, Plutus, etc
Tools for local deployment of blockchains for testing
Smart contract testing tools
Vulnerabilities and problems of blockchains and smart contracts in particular
API testing (HTTP REST, gRPC, etc.)

Conclusions

The world of blockchain is vast and continues to grow. It is not easy to get an initial knowledge about the topic because it is a mix of different areas: distributed systems, cryptography, finance, and others.

As there are not so many tools for testing blockchains, you may need to dig through a lot of whitepapers to get specific knowledge. And here is the beauty of blockchain (IMHO) - a short way from academic papers to the real-world implementation of the new concepts.

If you, like myself, are interested in blockchain testing - I prepared a list of resources about this topic. It is open-source - so feel free to use it and extend it with new blog posts and papers.

Do's and dont's of test automation interview

Oleksandr Romanov — Mon, 09 Aug 2021 08:07:11 +0000

Photo by Steve Halama on Unsplash

We all participate in job interviews. Not all interviews went well.
This blog post will share a few tips from my experience ( as a candidate and interviewer) on being a little better at test automation interviews.

Interviews - Do's

Do:

talk about the impact not about the tasks

Think about all your activities in terms of impact on business that you make.

If you have written 1000 UI tests, but they are flaky and nobody trust them - it is not an success. It is wasted time.

If you implemented 3 UI tests, integrate them into the build pipeline and the whole team gets a first automated feedback in matter of minutes after deploy - that's a success.

If you changed the quality "culture" and integrate new type of testing or "post-mortems" without writing a single line of code - it is even better.

Make your CV in such manner - concentrate on the impact.

always ask about the project's current stage, what's expected from automation, and the further plans.

It will help you understand whether the project is well established (and the only thing you will do is to add new or maintain existing cases) or it is a new one (so the processes and tools should be created or integrated).

If there is a proof-of-concept of the framework - ask about the choice of the technologies (and about the freedom to change the technologies).

It is better to know about the project/product more before joining it.

be ready to question, "Did you implement the automation solution / pipeline / process by yourself?".

Here you need to honest about the answer. If you got help from DevOps or developers - just say about it.

The candidate can't be a senior test automation engineer if he or she only fixed existing or add new tests (in the existing framework) for 3-5 years in a row.
.

be ready to question, "Why did you do this activity?".

In one of my interviews, I once asked a candidate how the testing process for the game was done. And the answer was, "Well, I just get a new build, install it and play in it." It is a good answer for a trainee, but not for an engineer with a decade of experience.

ask for feedback after the interview.

It will help you to get points where you need more attention and practice.

I know only a few candidates that asked me for a technical feedback. As a result they progressed much faster than other engineers.

always read the company (job) profile and prepare to interview according to that profile.

I failed one interview because I did not prepare myself for UI test automation questions (even though the company's leading product was websites and UI web technologies).

publish some code to Github or participate in open-source projects.

It is an excellent way to build a portfolio of your skills. Don't be shy that your projects or solutions will be too "simple." You can always enhance them after you get more experience.

Interviews - Dont's

Don't:

do not upload the whole testing framework code on Github (specifically including logins, password, connection details of the infrastructure) and share it with the interviewer.

Additionally, please remove package names or any other product details from the code.

do not hesitate to say that you have a bad connection during an interview call.

I had a screening phone call with a recruiter at one interview, and the connection quality was terrible. I asked, "Can you repeat, please?" again and again. So, as a result - I got a rejection with feedback about my low English level.

do not say to the interviewer: "I don't know how this piece of code works - I just copied and pasted it from StackOverflow, and it 'magically' worked."

Of course, it will show that you are an honest person, but from the other side, it will show that you didn't want to understand how the code works.

do not state ALL technologies you ever touched at work or pet projects in your CV.

Everything that you put in your resume - can be asked in the interview. If you wrote an "Open browser and search at Google" Webdriver test in Ruby, do not say you know Ruby. Keep the list of technologies short.

do not participate in the interview without a T-shirt :) even if the interview is a video call. And always - in video interviews, it is better to turn on the camera.
do not state that everything should be automated only via UI end-to-end level.

Explore other levels and types of testing and automation.

do not state that tool 'X' or framework 'Y' is the best choice for every possible task at work.

The products are different; the technologies are different too - so be open to new tools and opportunities.

do not say that code reviews are only for developers.

Treat automation code as production one in terms of clarity and quality.

Conclusions

It is not a complete list of do's and dont's for the interviews. I know that the more experience you get (as a candidate or interviewer), the more new tips you are collecting.

Good luck with your interviews!

Which tips can you add to this list?

Challenges in testing blockchain-based applications

Oleksandr Romanov — Mon, 02 Aug 2021 06:19:36 +0000

Photo by Terry on Unsplash

Blockchain and applications

Blockchain is a linked list of records called blocks.

The blocks are connected using cryptography: each block contains the previous block's hash and information about current transactions.

The blocks are immutable to change and are replicated across nodes in the blockchain peer-to-peer network.

The most famous blockchain implementation, for now, is Bitcoin. But blockchain is not limited only to cryptocurrencies.

As blockchain provides immutability, integrity, and transparency, it can store any data, not just transactions. Since the beginning of the Bitcoin release, many companies are constantly exploring new ways to apply blockchain to the current processes in the world.

The creation of smart contracts in Ethereum allowed simulating many business processes on the blockchain in a code (with either manual or automatic execution). As a result, now it is possible to build Blockchain-Based applications that allow communication with a given blockchain to store data, make transactions, or execute smart contracts.

All these blockchain applications deal with currency (tokens) or store and keep data on the chain. So we need not only build them but also ensure that the applications work as expected (from functional, performance, and security points of view).

In this blog post, I will describe the challenges in testing blockchain-based applications based on my experience and also a tremendous research paper of Chhagan Lal and Dusica Marijan - "Blockchain Testing: Challenges, Techniques, and Research Directions."

Architecture and testing

Modern blockchain-based applications do not implement blockchain "from scratch". They typically use blockchain networks as one of the components of the system.

Here is the sample architecture diagram of the blockchain-based application:

According to the diagram, the following components need to be tested:

unit-testing for business logic
integration between application and blockchain components (API)
functional testing of blockchain components (smart contract testing)
integration between application and data storage layer (API)
end-to-end testing (UI and API)

Few notes:

Integration between application and blockchain network is typically tested via API (REST, gRPC, command-line)
If application developers use an existing blockchain network, they may not test its internals (the same with database systems or messaging libraries).
Smart contracts verification include:
- functional testing (whether business logic works as expected)
- performance testing (check execution and time complexity)
- security testing (data protection and vulnerabilities)

As blockchain and smart contracts is a reasonably new technology, test engineers face with several challenges during testing blockchain-based applications:

lack of practices
lack of tools
immutability
performance
dependability

Lack of practices (development and testing)

Knowledge. To develop and test blockchain-based applications, engineers need to know about various domains (technical, non-technical, legal). But it is hard to find people with such knowledge on the market. E.g., in our company, test engineers need to know distributed systems, cryptography, blockchain, testing, programming, math, finance.

Variety. For now it is almost 40 different blockchains available on the market. You can't just "learn" one blockchain: there is an ecosystem of tools for each particular blockchain. Some blockchains use well-known programming languages like C++, Go, Javascript or Java for implementing smart contracts. Another - use their unique programming languages (Solidity for Ethereum or Plutus for Cardano) - so you need to learn them too.

Standards. Additionally, there is a lack of guiding procedures, standards, or strategies in designing, developing, and testing blockchain-based applications

Lack of tools

Libraries. As there are many programming languages for smart contracts - there is a lack of mature debugging / testing tools for these languages.

Automation. If you want to ensure that the application works as expected, you need to test the components (in isolation) and system (as a whole). Currently, there is a lack of deployment and automation tools, which allow to deploy and configure blockchain in a "test mode" within one machine's boundaries.

Environment configuration takes a lot of time. So the only thing now is to download the full node locally, synchronize it with other nodes and then test it via command-line or API. As a result, the configuration takes an enormous amount of time. If your application uses some wallet functionality, you also need to download and configure it.

Immutability

The blockchain is an immutable and open data structure (unless it is not a permissioned one). So as soon as any change is deployed to the network, you can't reverse it back.

Seriously. All the test data will remain on the chain forever :).

Moreover, the test data can be seen by any person via transaction explorers.

Test data need special care as it is available in a public blockchain. Make sure that you don't use any sensitive information in tests.

Validate smart contracts more and more before deploying them to the chain. Any error that slipped to the contract after the deployment can't be fixed easily - in some cases, you need to create another contract to send money back to the failed accounts (or even change the whole network after a hacking attack).

You can't delete the data from the chain. So, if your application needs to follow GDPR (right-to-be-forgotten in particular), you need to consider which data to store.

Performance

The performance of the blockchain-based application is highly tied with the performance of the blockchain components themselves. Some blockchains have a limited latency and rate of input transactions, so they need to be considered in the overall testing strategy.

Transaction validation time depends on a particular chain's consensus protocols - in some cases, it can take minutes or even more for storing one piece of data.

Real-world blockchain workloads are hard to predict due to volatility in blockchain popularity, hacking attacks, and other events.

Dependability

Fault-injections is a powerful technique in testing highly distributed systems. But due to the dispersed nature of the blockchain systems, it is hard to separate between system level, network level, and user-level fault loads. So there is a lack of tools in this area - mainly because each blockchain has its APIs and components.

Conclusions

Blockchain-based applications give a new set of challenges to the testing world. You need a whole lot of knowledge and skills.

But no worries. Imagine, somewhere like 20 years ago, nobody knows how to implement automated tests for web and mobile. There were times when Selenium Webdriver or Appium does not exist.

If we, as testers, want to succeed in the blockchain world and bring quality "to the table," - we need to dig into technical aspects of the systems and apply known (and maybe yet unknown) testing techniques and tools.

We will explore functional, performance, and security challenges and tools in more detail in the following posts.

Automation In Scala: Publish Allure reports with Github Actions

Oleksandr Romanov — Mon, 19 Jul 2021 09:57:41 +0000

Photo by Markus Spiske on Unsplash

In one of the previous blog posts, I showed how to add Allure reporting to the Scala-based automation project.

But generating reports only on the local machine is not enough. The main goal of test reports is to notify the team and the management about the issues found by automated tests.

That's why it is essential to share the reports using some notification channels. The channels can be various: emails, Slack or Skype notifications, etc.

In this blog post, I will continue my talk about Allure reporting: we will generate reports, push them to the server and notify users about test run results via email.

Build and test with Github Actions

Github Actions is a set of predefined steps for automating CICD workflow for the project. It is defined in simple YAML format.

Action can have one or more triggers: on push, pull request for a particular branch. Special triggers allow to run jobs manually (workflow_dispatch) or after getting a special event from another repository (repository dispatch).

In our case, we will add a trigger for running workflow by manual execution.
Just add .github/workflows folder structure at the root of the project and add a new .yml file with the description of the flow:


name: Execute Test Run

on:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up JDK
        uses: actions/setup-java@v1.4.3
        with:
          java-version: '11.0.8'

      - name: Compile
        run: sbt compile

      - name: Test
        run:  sbt "testOnly * -- -n Smoke"

Here we define test job for our project that will be executed on ubuntu build machine:

it will perform checkout for branch
then, it will add JDK to build machine
then it will compile the project and execute the tests

As we use sbt as a build tool, then we run test job with filter for specific tests

Allure server setup

Allure server is an open-source solution for hosting test reports.

You can download the project from GitHub and execute it on the local machine, but the best case is to have a dedicated server for reports - e.g., some instance in the cloud.

For configuration, you need to download Docker image and start the server:


docker pull kochetkovma/allure-server:latest
docker run -p 8080:8080 kochetkovma/allure-server:latest

After you start the server, it will be available at http://127.0.0.1:8080.

Ensure that the Allure server is accessible for calls from Github, as we will call server API for saving reports.

Reports publishing

We will use github action from @Xotabu4

Under the hood, this Github action will make an archive from the allure-results folder and store it at the Allure server.

As a result, this action will return the URL to generated report, stored on the server.

If you want to use the report URL in further steps (e.g., notification), please add an id property to the action with some name.

Email notification

Email notification is executed via action-send-mail.

Here is an example of action configuration:

Few notes:

Username and password for connecting to Gmail should be added as SECRETS for repository
As email action connects to Gmail for sending emails - make sure that security policies allow to do it for external resources
You can use Allure report URL from the previous action as steps.allure-report.outputs.report-URL
Date variable from Get-Date step is used as steps.date.outputs.date
Recipients are configured as a comma-separated list
Subject of the email notification will include date and time of the test run. In order to get it, we will use special action:

Executing action

Navigate to Your repo on Github / Actions tab, select "Execute Test Run" workflow and click Run Workflow.

As a result, you will get an email with a link to the Allure report (saved on the Allure server).

Conclusions

As you can see, Github Actions allow the creation of automated testing workflow in less than 50 lines of code.

If your end-to-end tests are a part of the solution - add these actions into the workflow.

If your tests are stored in a separate project - you can use repository_dispatch for subscribing to events from other repositories in the build pipeline.

You can find the source code of the action at the repository

Should You Use Contract Testing?

Oleksandr Romanov — Mon, 12 Jul 2021 09:53:02 +0000

Photo by Cytonn Photography on Unsplash

Almost any tester involved in testing microservices-based systems has heard about the magic contract testing.

But what exactly are contract tests? Should you use it in your project?

In this blog post, I will talk about the case when contract testing can help. I will also compare contract testing with other types of tests and share my thoughts on using contracts in your application.

The story

Let me introduce two characters of our story: Dave (The Developer) and Tina (The Tester). Both of them work in one Scrum team delivering a critical piece of software for a conference application. The whole backend contains hundreds of microservices.

Their task is to provide the functionality to get bonuses for a given user. Dave needs to implement a new API call from the User Conference service to the Bonus Conference service to retrieve the bonus data for users.

The developer way

As a good developer, Dave wants to cover new functionality with tests. The best possible way to do it - is unit testing.

So Dave implements many unit tests for checking feature functionality on the User Conference Service. All external calls to Bonus Conference service or any other resource are mocked (so Dave is responsible for defining the response values).

Dave creates more and more unit tests; the code coverage tool says that it is covered more than 80% of the new code.

Dave finishes his work with the confidence that the tests catch-all error cases and bugs. All tests are executed locally within seconds.

The tester way

After Dave's change is deployed to the staging environment, Tina starts the testing. And it turns out that the new functionality does not work at all.

The root cause of the problem is that another developer changed the Bonus Conference service API, and Dave was not aware of it at all. All unit tests check the service with the older version of the API.

Tina, as a tester, catches the broken change only after end-to-end testing of the functionality.
After the root cause analysis, Dave and Tina decide to cover the broken case by end-to-end automated test. It will detect the broken API issue when the User Conference service will be deployed to the test environment.

So from the one hand, Dave has many unit tests that provide good coverage, fast feedback, and runs on a local machine.

On the other hand, the end-to-end test will provide an additional check of broken integration, but the feedback time will be slow: all services must be deployed to the test environment first.

Given the number of services (hundreds) and updates per day (thousands or more) - each developer should wait for an entire deployment to find the end-to-end test results.

Maybe there is another way of getting information about broken integration? It turns out that it exists.

Contract testing to the rescue!

Contract testing is another approach to test integration between two counterparties.

These two sides can be services (communicates via HTTP REST, messaging, or gRPC calls), client and server, various third parties.

In contract testing terminology, the side that makes the request is called the Consumer; the side that responds is called the Producer.

The main idea of contract testing is to provide fast information about broken integration without deploying services to the environment. Imagine it like an end-to-end test that delivers the results with a speed of unit test.

Instead of creating separate unit tests on the User Conference and Bonus conference services, Dave will start with the contract definition.

The contract defines what the User Conference service will request and what it expects to get in response.

Here is an example of the contract, using Spring Cloud Contract library:

Then both services should implement contract tests for each contract. After contract tests are implemented, the most exciting part begins.

If the developer wants to change to Bonus Conference service API (the Producer), he needs to change a contract. Otherwise, the Bonus Conference service will not be built and compiled locally due to failed contract tests.

After the developer changes the contract, all consumers will be notified about the change. So they need either approve the change (handle it in their services) or comment that this change breaks the integration.

Test types analysis

Let's compare three types of tests: unit, contract, and end-to-end API.

Execution. Unit and contract tests are executed locally on the developer's or build machine, but end-to-end tests require services to be deployed to some dedicated environment.
Feedback time. Feedback time of failed tests is fast for unit and contract tests. Of course, contract tests require additional seconds to generate tests from contracts, but the overall execution time is almost like for unit tests. As for end-to-end tests, we need to wait: wait for service to build and compile; wait for services unit and contract tests execution; wait for the deployment of the services to environment; wait for test execution.
Setup complexity. For writing unit tests, you need to include a unit-testing library in the project, and you are ready to go. In case of contract tests, you need to set a contract test repository and build an automatic workflow for consumers and producers. For end-to-end tests, you need to develop a deployment pipeline and tests themselves.
Change tolerance. Unit tests are build only on one developer's assumption on how another service should work in a given moment. If something is changed, unit tests will always be "green." On the contrary, contract and end-to-end tests will fail in case of broken integration changes happen.
Functionality verification. Both unit and end-to-end tests verify functionality. The main goal of contract tests is to check only the format of communication between two parts.
Team collaboration. To implement unit tests, the developer does not need to communicate with other developers. Contract and end-to-end tests push the developer to communicate and to think about the impact of changes.

Should I use contract testing?

Contract tests are not a substitution to the unit or end-to-end tests. It is just one type of test that can be performed.

If contract testing workflow is implemented correctly - it is possible to minimize end-to-end tests and speed up the development process.

So the crucial question remains: "Should I use contract testing?"

You will not get a value from the contract testing if:

the coverage of unit tests is low
the number of end-to-end tests is close to zero
there is a lack of skills for setting up the project and writing the contracts
there is a lack of dedication to change a process of delivery
the communication between teams is broken (e.g., blaming each other for integration issues)

Conclusions

Contract tests are not a "silver bullet" which provides only the benefits.

It requires many efforts for configuration; it requires even more efforts to change the developer "culture." With contract tests, you can't make hidden changes, silently push it and close the Jira as soon as possible.

And most important: contract testing forces you to think more about the consequences of each change that goes to production.

Have you already tried contract testing for your projects? Did you get any benefits?

Modern Problems With Test Automation

Oleksandr Romanov — Mon, 05 Jul 2021 12:40:48 +0000

Photo by kevin laminto on Unsplash

Brave new world

Test automation is everywhere. Almost every job description in the software testing industry requires knowledge of programming and automation at some point.

If you don't have these skills - please go to numerous courses and learn the skills. Now, as soon as possible.

Otherwise, you will not get a good job. Or you will not be promoted. Or you will not be an SDET (Software Developer In Test).

But do we get the value from test automation? Do we correctly implement automated tests for our projects? Are there any problems with that?

In this blog post, I want to highlight a few problems with test automation. Some of the issues I faced personally, other - I know by talking to other engineers.

Don't want to see the trees for the forest

Concentrating only on the UI/API end-to-end level.

"If they want test automation, we will create a bunch of UI end-to-end tests and get immediate results! Everybody will be happy! Bugs will not slip into the release!"

"Let's use this new fancy record-and-playback tool, and we will be able to create automated tests even without programming knowledge."

"Let's use BDD (Behaviour Driven Development) approach, and the team will create end-to-end UI tests which business people and developers will use! I have not asked them yet, but they will use these tests for sure!"

"Unit, component, integration tests... Oh, boy! It is too technical, and I am not a developer! I want to fix UI tests day by day."

"I heard at the conference that this tool is popular, so I purchased a license for a year! Now we as a team need to find out how we can use it."

"Our UI automated tests are green - why do we still have bugs?"

Thinking fast and small

Implementing quick and destructive solutions, and then moving to the next project.

"Now I am going to implement a test framework with this fancy new library X and move to another project ASAP!"

"If I use this new library that just been released a month ago - I will be able to put it into my CV and request more salary!"

"Who cares that for test execution, you need first to install a hundred dependencies, update 39 different files, and re-compile half of the product?"

"Why should I write README for the test project? Anyone can understand the solution and quickly start implement the tests!"

Scratching a technology surface

Lack of focus on deep technology and architecture knowledge.

"I don't know how the system work - for me, it is only a set of web pages or mobile app which is somehow works (by magic)."

"Subsystems, abstractions, load-balancers, HTTP calls, messaging, replication? Wow - it is too much for me. I will write the "Open Google page" test one more time using a bright and shiny new framework."

"Wow, I just clicked something, got some scary error, make a screenshot and throw it to the developer."

"Why has a bug happened? How can we prevent it? - Oh, I don't know, the developer did a fix, and now it works. Somehow. I don't care."

So what?

For junior test automation engineers, it is entirely normal to ask all of such questions. It is normal to concentrate on the UI tests only, do not understand bug stack traces, and do not dig into the technology.

But if you are a senior engineer now and still see many such questions asked, here is what we can do to make the situation a little better.

Here what you can do:

Learn the programming language and technology stack of the system you are working with. Which framework do developers use, which database? How is the system deployed? How is your data replicated across the system? What are third parties to an application under test? The more you know how the system works, the faster you can investigate and fix the failures. And the better tests you can create.
Investigate more about different levels of testing: unit, component, integration, contract. What are their pros and cons? Why can't you use such tests in your project? What blocks you from adding such tests?
Discover the root causes of the issues together with developers and DevOps engineers. Is it possible to find the bug before the UI tests caught it? Which actions can you do now to prevent such issues in the future? Where do you have "holes" in code coverage?
Do not consider your job as a "fast fire-and-forget consultancy. (Unless you are a consultant)" Always ask stakeholders why they need to have test automation. Which value do they want to get from such tests? How can you help them to release faster and more reliable? What is the product vision for upcoming years, and how automation fits in?
Explore how other companies solve issues with test flakiness, stability, scalability. In most cases, your application is not a unique - so you can find out that your current struggles have already been solved by some open-source tool that you can use.
Do not try to use every fancy tool on the market. Start with careful evaluation and compare different tools in the segment. Do not believe that automated tests can be recorded once and be executed forever. Maintenance always matters!
Always think about which impact do your engineering efforts make on the product. Is it make the life of other developers easier? Does it help to investigate issues faster?

Conclusions

Every team member can be a source of automation problems. Maybe it is a test manager who wants to change a process and tools. Perhaps it is a developer who doesn't like to share information on how the product works internally. Or it can be a test automation engineer who wants quick results without any effort.

Anyway, it is up to you to identify and fix such problems. Good luck in fixing it!

Have you ever faced any of these problems?

Getting Started With Testing Microservices

Oleksandr Romanov — Mon, 28 Jun 2021 07:38:17 +0000

Photo by Ryoji Iwata on Unsplash

Microservices testing: from the theory to practice

In this blog post, I will share resources that help you to be more prepared to testing microservices-based systems.

My background is Java and Spring Boot. Many tools and libraries that I will share are from the Java world. But you can find analogies in other technology stacks as well.

I will split resources into the following sections - from the theory to practice:

Learn the basics
Explore technology stack
Build test strategy
Learn tools
Investigate contract testing

Learn the basics

The first thing is to understand microservices architecture and how it differs from more traditional monolith systems.

The following resources will help you to build initial information:

Building Microservices book by Sam Newman
What are microservices video by freeCodeCamp
Microservices overview by Martin Fowler
Microservices at IBM Cloud
The What, Why, and How of a Microservices Architecture by Jetinder Singh

Explore technology stack

Resources (Spring Boot):

Spring Boot - project documentation
Learn Spring Boot - Rapid Spring Application Development course on Udemy
Spring Test library overview
Testing WEB APIs at Spring Boot

Of you extensively use virtualization tools, like Docker - learn it.

Pay attention to how to start a single microservice in Docker container on your local machine (together with its dependencies: SQL, NoSQL databases, messaging systems, etc.):

If you are not from the Java world - ask your fellow developers about technologies and frameworks they use for building microservices in your organization.

Do not hesitate to ask which testing tools, practices, and guidelines they already use.

Build test strategy

Classic approach with unit, integration, and end-to-end testing can't be applied to microservices "as is." That's why approaches to testing need to be reworked and adapted.

Resources:

Testing Strategies in a Microservice Architecture by Toby Clemson
The Practical Test Pyramid by Ham Vocke
Testing Microservices: an Overview of 12 Useful Techniques by Wojciech Bulaty and Liam Williams
Testing Microservices, the sane way - very big and highly informative article by Cindy Sridharan
Descending the Testing Pyramid: Effective Testing Strategies for Microservices slides by Chris Richardson

Learn tools

Tools and libraries can greatly simplify testing and automation and boost developer productivity.

Resources:

Wiremock for mocking external API (API virtualization)
Swagger for API visualization, research, and testing. Ideally, each microservices should provide an API for public and admin usage
In-memory databases will help you to prepare service dependencies faster and make them more reliable for unit testing
Testcontainers. The incredible library automatically runs service dependencies (databases, messaging, other services) in a separate docker containers. As a result, you will have a "real-world" setup for service component tests - but with much more control over dependencies.
Rest Assured for API test automation. It is just one of the possible tools: do not hesitate to try something else and find a tool suitable for particular purposes

Investigate contract testing

When the number of microservices is increasing to hundreds or thousands, the problems of integration arises:

if I deploy this API change - which microservices will I break?
which services depend on each other?

Contract testing allows building another layer of fast and reliable tests, preventing integration issues as soon as possible in the development workflow.

Depending on the technologies used on your project, you may prefer:

Spring Cloud Contract. It works best with Java and Spring ecosystem (but not limited to it)
PACT tool. It supports multiple programming languages out-of-the-box

If services use gRPC protocol for communication - you can use PACT for contract testing as well.

Resources:

What is contract testing by PACT developers
An introduction to contract testing blog post series by Bas Dijkstra
Introduction to contract testing with Pactflow video series by PACT developers
Practical Contract Testing with Spring Cloud Contract - my talk at TestCon Europe
Spring Cloud Contract course
gRPC course

Conclusions

Microservices architecture is not a "silver" bullet in software development. The benefits also come with drawbacks.

For tester's point of view - testing microservices is like a testing system of systems: you start with single service verification in isolation, then move to integrate it with dependencies and other services, then you move to end-to-end API and UI tests as for any application.

The biggest challenge here is that the previously described testing process needs to be applied to many different services. It is not fast and scalable to do it manually - so high investment in automation is a prerequisite to any successful microservices testing effort.

This list of resources is not complete - it is just the best ones that helped me a lot in the past.

What is your favorite book or blog post about microservices?

Automation in Scala: API tests for gRPC service

Oleksandr Romanov — Mon, 14 Jun 2021 06:32:34 +0000

Photo by Nick Fewings on Unsplash

What is gRPC?

gRPC - is a modern Remote Procedure Call framework created and popularized by Google but now is under CNCF license.

The main idea of the framework is to provide for clients, written in various languages, a possibility to execute remote calls to the server as if it is a local call.

Picture from grpc.io docs

The core concept of gRPC is that all communication between server and client is based on Protocol Buffers defined in proto files. The server will implement and support this interface, and the client will generate and use stub code.

More about gRPC and concepts - here.

Introducing: Greeter service

gRPC service example is taken from the official akka-grpc guide.

server-grpc-example repository contains a proto file with a service definition and its implementation. You can execute it on a local machine by running:

$ sbt runMain io.grpc.examples.helloworld.GreeterClient

By default, the server will start at localhost:8080.

Protobuf definition for the service is the following:

syntax = "proto3";
import "google/protobuf/timestamp.proto";

option java_multiple_files = true;
option java_package = "example.myapp.example.myapp.helloworld.grpc";
option java_outer_classname = "HelloWorldProto";

service GreeterService {
  rpc SayHello(HelloRequest) returns (HelloReply) {}

  rpc ItKeepsTalking(stream HelloRequest) returns (HelloReply) {}

  rpc ItKeepsReplying(HelloRequest) returns (stream HelloReply) {}

  rpc StreamHellos(stream HelloRequest) returns (stream HelloReply) {}
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
  google.protobuf.Timestamp timestamp = 2;
}

As you can see, Greeter Service has four methods:

SayHello() method for unary calls: a client sends a single request to the server and gets a single response back
ItKeepsTalking() method for client streaming: the client will send a bunch of messages to the server and, after that, will wait for a single reply back
ItKeepsReplying() method for server streaming: the client will read a stream of messages from a server until there are no more new messages.
StreamHellos() method for bi-directional communication: the client and the server send a stream of messages to each other.

Automate by consuming proto files

Test automation process for gRPC based service consists of the following steps:

Add sbt plugin for Akka gRPC - create plugins.sbt file with the following content in the repo/project folder.
```
addSbtPlugin("com.lightbend.akka.grpc" % "sbt-akka-grpc" % "2.0.0")
```

Add dependencies to build.sbt and update the project.

enablePlugins(AkkaGrpcPlugin)

libraryDependencies += "org.scalatest" %% "scalatest" % "3.3.0-SNAP2" % Test

Get all proto files from the server repository. In our case, it will be just copying hello_world.proto file from server repo and pasting it to src/main/protobuf directory.

As an improvement, proto files can(and should!) be automatically downloaded from the server or central proto file repository to all client repositories. It prevents the client from synchronization errors as the development of the server functionality evolves.
Generate gRPC clients from proto files. As our build system in sbt, you need to execute the command
```
$ sbt compile
```
in Terminal or choose it from sbt command list in IDE.
As a result, all clients generated from proto file can be found at /target/scala-2.13/akka-grpc/main folder.

Add Matchers and ScalaFutures traits to test class definition.

class GreeterServiceApiTest extends AnyFlatSpec with Matchers with ScalaFutures {

Configure default waiting timeout for responses.

implicit override val patienceConfig: PatienceConfig =
    PatienceConfig(timeout = Span(3, Minutes), interval = Span(10, Millis))

Add configuration for gRPC client.

val host = "127.0.0.1"
val port = 8080

implicit protected val system: ActorSystem = ActorSystem("api-test")

val clientSettings: GrpcClientSettings = GrpcClientSettings.connectToServiceAt(host, port).withTls(false)

val client: GreeterService = GreeterServiceClient(clientSettings)

Implement the tests

"Greeter Service" should "handle unary requests and response" in {
    val name = getRandomValue
    val response = client.sayHello(HelloRequest(name)).futureValue

    response.getTimestamp should not be null
    response.message should be(s"Hello, $name")
}

"Greeter Service" should "handle client streaming" in {
    val data = getRandomList(3)
    val response = client.itKeepsTalking(Source(data.map(HelloRequest(_)))).futureValue

    response.timestamp should not be null
    response.message should be(s"Hello, ${data.mkString(", ")}")
}

"Greeter Service" should "handle server streaming" in {
    val name = getRandomValue
    val responses = client.itKeepsReplying(HelloRequest(name)).runWith(Sink.seq).futureValue

    responses should not be empty
    val messages = responses.map(_.message).toList
    all(messages) should not be ""
}

"Greeter Service" should "handle bi-directional streaming" in {
    val data = getRandomList(3)
    val responses = client.streamHellos(Source(data.map(HelloRequest(_)))).runWith(Sink.seq).futureValue

    responses should not be empty
    val messages = responses.map(_.message).toList
    all(messages) should not be ""
}

Pay attention that we can't get a response straight away - we need to wait until Future completes and then get the data by executing .futureValue.

Matchers library provides a fluent api for writing assertions - for single values and collections of elements.

Execute the tests either from IDE or from Terminal
```
$ sbt test
```

Conclusions

Automation of gRPC-based services is a little bit similar to automating SOAP-based services. In both cases, you have some utilities for generating clients as a code (ws-consume for SOAP and akka-grpc or scalapb for gRPC).

After you generate a client - an overall approach is the same as for any API tests - firing a request with test data and asserting responses.

gRPC brings a bit more complexity with client, server, and bi-directional streaming - but you can handle it without any problems using futures and collecting all responses in Lists.

As always, all code is available in github-repo

My biggest testing regrets

Oleksandr Romanov — Sat, 12 Jun 2021 10:58:13 +0000

Photo by Brett Jordan on Unsplash

Testing regrets

Ministry of Testing Bloggers Club offered an exciting topic for June - "Your biggest testing regret."

I started my professional career in September 2011. My first job was as a QC Engineer, but I soon switched to test automation and software development. As of 2021, I worked in different companies - from banking and finance to gaming with various technology stacks.

I am not a fan of regrets. So in this blog post, I want to tell more about lessons learned than just simple regrets.

Personal development inside the company

During my first years in university and my first years in a full-time job, I hoped that somebody would "magically" create a strict development path for me.

"Just learn this and that, practice this, and you will become a middle/senior/lead test/automation engineer."

Unfortunately, it did not happen in university, so I needed to learn a lot by myself. So it did not occur at work.

What I learned is that the only person responsible for your personal development - is you.

It is not your lead, manager, CTO, or career advisor. It is YOU.

You need to think about what you want from your current and future career: whether it will be testing, development, automation, leading people, or starting your own company.

You need to think about which companies do you want to work with in the future.

You need to create a personal development plan for yourself. Analyze which skills do you need to have. Which frameworks or programming languages do you need to master.

Think about it as an RPG game.

Create a mind map with skills that you have, and want to have. It can be a path for the upcoming 3-6 months or a couple of years.

Set goals and start to learn new things. Learn from books, courses, videos, blogs. Learn for colleagues.

You can discuss your plan with your manager. Ask him/her whether you can apply new skills in your current project. Ask about future opportunities within a company. If you see that a company can't (or don't want) to support you in your development - consider changing your job.

Take responsibility for your own career "story," and you won't regret it.

Test engineering is considered as non-technical

In many work environments where I worked, testers or automation engineers are considered "not-so-technical" engineers. It is a huge regret for the whole industry.

Testers are just clicking buttons and file bugs. Testers just sent a message "Something is not working" without attaching any logs, environment information. Testers do not provide clear steps to reproduce the issues. Testers can't even give initial thoughts on the root cause of the bugs.

I also thought in such a way for some time.

The root cause of such a problem is because most QA courses do not provide a solid technical background on how systems work. The primary purpose of such classes is to teach you how to follow predefined test cases and file bugs. The most "technical" thing is firing a few HTTP calls manually or writing simple WebDriver tests for the Login page.
And after all, what the heck do you want to learn from scratch after 1-2 month courses?

As a result, most testers want to move to the development roles as soon as possible. It can be software development, managing positions, business analysts, etc.

If you stay in testing for more than ten years, you feel like a "stranger" person.

All of this is why testers themselves do not know or feel that their work can be technical.

But magic can happen.

If you invest in developing your technical skills, you, as a tester, can see that the overall picture is changing.

It changes when you go to the developer with an initial investigation of the root cause. Or with possible solutions along with the trade-offs.

It changes when you describe a possible "what if" scenario during a discussion of future deployment strategies.

It changes when you highlight possible issues with scalability or reliability when you talk with architects.

And after all, it changes when you clearly show the risks to managers and stakeholders, supplying it with business metrics and data - not just words.

Developing technical skills is challenging. It requires time and dedication. But it pays off.

Testers are engineers. They can perform academic researches. They can apply programming to create the tools which benefit the team. They can do a complex analysis of the system to find "weak points."

That's what engineers can and should do.

Paper review: Simple Testing in Distributed Systems

Oleksandr Romanov — Tue, 01 Jun 2021 05:05:57 +0000

New format

I enjoy reading research papers. Here you can get an overview of the latest concepts and ideas in the field. You can get an overview of the newest tool and approaches from the top companies.

You can also get the results of the academic research performed by scientists on a big chunk of data. Reading a research paper is like reading a book, but in a much smaller format and with a lot of new information per page.

Recently, I started to look for and read research papers about testing in the various types of distributed systems.

That's why I want to start a series of posts with a common title: "Paper Review". Here I will share the most exciting papers about quality that I found.

Distributed systems

What is a "distributed system"? In its simplest definition, it is:

group of computers that
work together on a given task and
communicate with each other by passing messages and
appear to a user as a single machine

It turns out that distributed systems are everywhere.

The Internet is in some way a distributed system of computers that provide information to the user.

Web / Application monolithic backend is a distributed system in a case when deployed onto multiple servers.

Backend based on microservices is also distributed system. Dozens and hundreds of microservices are working together to provide some results. And each microservice also can have multiple copies.

Blockchains are distributed systems. Underneath it is a system of many computers sending each other information about a newly added block of transactions.

Almost any modern application contains parts distributed in nature: storages, message queues, load balancers, etc.

Testing in distributed systems

If you are working in IT, you know that testing can be challenging and complex. The main complexity is that the number of test cases and the number of potential issues in the system are almost infinite.

It even gets more challenging when the system is distributed.
You need to check application functionality.

But you also need to check how data is replicated between different nodes in the storage system.

You need to test any synchronization issues between nodes located in different timezones (e.g., providing user data from other CDNs).

And the main thing - you need to test how the system will behave when one or more nodes are down (restarting or broken).

It is complicated for a couple of nodes, but what can we do if the number of nodes is 100s or 1000s?

An overview

The paper "Simple Testing Can Prevent Most Critical Failures" by Ding Yuan et al. gives a good view of which failures can happen in real-world production systems.

The authors performed an analysis of critical failures of the five distributed systems: Cassandra, HBase, HDFS, MapReduce, and Redis.

During an investigation, they reveal a lot of interesting findings. Here are the most interesting ones:

The majority of the failures require more than one input event, but it requires no more than three events in most cases. 50% of the losses require only two nodes.

This finding says that you don't need to simulate a large production cluster with thousands of nodes for reproducing the failure or for testing. You need only two or three nodes. It simplifies the approach to building test environments.
The majority (74%) of the failures are deterministic. It means that it is possible to get the correct order of multiple events to simulate the failure. Random-based test generation can help, but an order of events is defined and can be transformed into a test case in most cases.
76% of failures print log error messages and have logged inputs. So they can be clearly understood and reproduced.
The unit test can reproduce 77% of the production failures! Do not underestimate the power of unit testing. Integration and end-to-end testing can reveal these problems - but they didn't find them as these bugs slipped to production.
92% of the failures are due to incorrect error handling (fatal and non-fatal). A lot of error handlers were too general or missed some extra corner cases. In some cases, it even contains TODO and FIXME comments. It means that error handling code should be analyzed by external tools and other developers and testers to find potential gaps.

Conclusion

As you see, distributed systems add additional complexity to testing due to their scale. But it turns out that the most trivial testing approaches like unit testing of error-handling cases - can prevent the majority of such issues.

We, as quality engineers, should help developers to identify such potential gaps in coverage. It can be by a manual investigation; it can be by improving existing code coverage scanners.

What was your most exciting bug found in a distributed environment?