DEV Community: IntexSoft

Container Orchestration: What Is Kubernetes

IntexSoft — Wed, 23 Dec 2020 07:47:17 +0000

First of all, i would like to thank my colleague Dzmitry Iliushyn for helping with this article.

With the boom of horizontal scalability and microservice architecture, the world observed the “container deployment era”. Containers have been in use for more than ten years. Today at least a quarter of leading IT companies use container solutions in a big production, and this number is likely to grow further.

Many market solutions provide container runtimes and orchestration, such as Docker Swarm, Mesos and others. However, more than half of the users choose Kubernetes as an infrastructure standard. Here we'll talk about the Kubernetes as a widespread and rapidly-developing solution.

Why containers?

Let’s start with container peculiarities and advantages first.

As Docker developers would say: "a container is a standard unit of software that packs an application with all the dependencies it needs to run - application code, runtime, system tools, libraries, and settings." Compared to Virtual Machines, containers, in essence, are not full-fledged independent OSes, but configured spaces isolated within the same OS that use the host's Linux kernel to access hardware resources, as well as host’s memory and storage. Therefore, containers require significantly fewer resources to start and run, which positively impacts the project's performance and budget.

Lightweight, high-speed of startup and great performance, delegation of hardware and OS maintenance to the provider (in case of -as-a-Service solutions), are the advantages of containers, which reduce the costs of development and maintenance of applications making solutions based on containers so attractive for businesses.

As for advantages for technical specialists, containers allow them to pack an application along with its runtime environment, thereby solving the problem of dependencies in different environments. For example, differences between software package versions on the developer's laptop and the subsequent staging or production environments will sooner or later lead to failures. So it will at least require efforts to get with them: to analyze and fix bugs edged into production. Using containers eliminates the "everything-worked-on-my-machine" problem.

Containers also reduce app development time and simplify its management in production due to:

easy setup and changing the configuration;
versioning the configuration along with the application code;
creating an easily reproducible environment for implementing CI/CD processes;
convenient orchestration tools allowing to scale the infrastructure quickly. Besides, the absence of binding containers to the hosting platform gives tremendous flexibility when choosing or changing a provider. One can run them without fundamental differences on a personal computer, bare metal servers, or cloud services.

Containers are a great way to bundle and run applications. In a production environment, DevOps engineers need to manage the containers that run the applications and ensure that there is no downtime. To withstand the estimated load, the number of application containers should be certain or not less than minimum required. So here some mechanism is required which will start new copies of the app containers if load will become higher or some of them fail for any reason. And this process is easier when such a behavior is handled by the system like an “orchestrator” for these containers. All this leads to Kubernetes.

About Kubernetes

Kubernetes (also known as K8s) is a platform implementing orchestration of containers in a cluster environment. Previously, it was mainly used for Docker containers as they were highly popular and wide-spread . But, in the latest Kubernetes versions, Docker is listed as deprecated while Container Runtime Interface (CRI) is stated as the native container mechanism.

Kubernetes significantly expands container capabilities, making it easier to manage deployment, network routing, resource consumption, load balancing, and fault tolerance of running applications. This platform is now widely offered by Cloud providers within the -as-a-Service model. At the same time, nothing stops you from installing it on a set of physical or virtual machines whether in cloud or on-premise.

Kubernetes is a new step in the IT industry that allows simplifying the delivery of apps including the environment where these apps will run. With Kubernetes, it is possible to abstract the description of this environment from the hardware configuration.

K8s allows us to create a cluster based on several physical or virtual machines. This cluster works as a single environment and consists of two types of nodes:

Master nodes that manages the cluster (for high availability best practice there should be at least 2 master nodes);
Worker nodes that are the machines to run the applications (2 or more as for full-fledged cluster depending on load).

When you deploy your applications in K8s, you alert the master to launch app containers. The master node schedules containers to run on the working nodes. Each working node communicates with the master. For that Kubernetes has the API, which can be used for direct management using REST calls as well.

To get started with Kubernetes developers can use a lightweight implementation - Minikube. Minikube is a tool available as a cross-platform solution. It creates a simple single-node cluster on a local machine.

To work with a Kubernetes cluster and to run an app in such an environment, one needs certain instructions to determine the desired state of the app. Such instructions are Kubernetes Manifests that describe what and how you want to run the containerized code in a cluster. Basic manifests could be relatively simple. They are code in YAML/JSON format, so for a developer it would not be a problem to write a simple manifest, also Manifests can be handled using the GitOps model. But when we are moving from simple manifests in development to production, the problem is that the level of K8s abstraction is so high that one should have a good understanding of how things work under the hood of a cluster.

Kubernetes has a lot of Kinds that describe and manage the behavior of containers inside a cluster. K8s itself is primarily a Container Orchestrator and therefore its parts are containers as well. This fact leads us to 2 main peculiarities:

The components of the platform do not affect each other. If one goes down, then all others continue to operate.
These components work according to a pull model. There is no central component, a commander who rules everyone. Each component carries out and performs its own function.If the component fails, then it's actions will not be performed.
Kubernetes model is focused on maintaining the desired number of working containers: it restarts the containers that go down, or don't respond to a user-defined health check, replacing them.

Core Kinds of Kubernetes

Node
A node is a component part of the Kubernetes cluster. The Master node (there can be several of those) controls the cluster through the scheduler and controller manager, provides an interface for interacting with users through the API server, and contains the etcd repository with cluster configuration, statuses of its objects, and metadata. The working nodes are intended exclusively for launching and working with containers; for this, two Kubernetes services are installed on it - a network router and a scheduler agent.

Namespace
A structural object that allows you to divide cluster resources between environments, users and teams.

Pod
This is the smallest unit in Kubernetes - a group of one or more containers that work as one service or application, assembled for joint deployment on a node. It makes sense to group containers of different types in a Pod when they depend on each other and, therefore, must run on the same node to reduce the response time when they interact. An example: containers with a web application and a caching service.

ReplicaSet
An object that describes and controls the correspondence of the number of Pod replicas running on the cluster. Setting the number of replicas to more than one is required to improve the application's fault tolerance and scalability. Deployment kind provides more capabilities and is used more often for apps, but in some particular cases ReplicaSet is enough.

Deployment
An object that declaratively describes Pods, the number of replicas, and the strategy for their replacement during update. Deployment allows updating different Pods according to declared rules. This makes the update of services or applications that have been released as much smooth and painless as possible.

StatefulSet
StatefulSet allows you to describe and save Pods' state: unique network address and their disk storage while restarting, thus implementing the stateful application model.

DaemonSet
It's a kind that provides control over what will be launched on each (or several selected) node according to a specified Pod instance. It is used to run cluster storage, logs collecting or node monitoring.

Job and CronJob
Kind that launch the specified Pod once (or regularly on a schedule in case of CronJob) and monitor the result of its execution.

Service
A virtual kind for publishing an application as a network service, which also implements load balancing between application Pods.

CRD
K8s is not limited only by its basic functionality, but can also be extended with Custom Resource Definitions (CRD). CRD allows us to inject or even “invent” new custom kinds in addition to out-of-the-box ones.

Pros and Cons of Kubernetes

The K8s in essence is a great abstraction tool: there is no need to think deeply about technologies, hardware resources or its maintenance, etc. Let’s look a bit deeper.

+Abstraction from hardware
First of all, there is no need to deep dive into hardware characteristics and parameters, despite the project itself not requiring some specific needs. In case of any custom solutions, developers or DevOps engineers indicate these custom adjustments in the manifests or probably even add checks for them.

For example, if your project uses a database and it’s not a cloud database -as-a-Service, you should have one or several nodes a bit more improved then the others (they should have more memory and fast storage) and you should just “tell” via manifests for database pods to be run on exactly these improved nodes.

Moreover, you are able to limit all the needed resources (RAM, CPU, etc) yourself. The lack of processing capacities can be compensated by cluster monitoring plus node scaling. If talking about the Cloud, this process can even be automated with a Cluster Autoscaler which scales the target cluster up or down, so it can satisfy any changing demands of the workloads on top.

Autoscaling is a great solution when you don’t know exactly what will be the load and how many resources the app needs to keep such load. As an example: selling the tickets for a football match, when the requests traffic increases within a short period of time. So with Cluster Autoscaler, the cluster expands automatically to cope with the load and returns back after load decrease.

+Resource isolation
With the help of namespaces, you can isolate several projects or different stages of one project - non-prod and prod environments - from each other in one cluster. With RBAC you can also configure restriction of access to valuable resources based on the role the user holds. All this provides us with predictable application security.

+ Safe update of microservice systems
All we know, that microservice architecture is widespread now. For example,if we have a website divided into modules that operate separately, the failure of one will not affect the whole system. Moreover, there is a possibility to update each separate module with Deployment so that any problems with the update do not engage the whole audience but nobody or a small part of them. And, eventually, if something goes wrong, versioning will help us to roll the update back.

+Uniformity
There is no actual OS dependency, you just have an integrated interface you interact via standard commands or API with.

+Integration with third-party software
Integration with monitoring systems, CI/CD systems, secret storages and many more . As for the secrets: if you don’t want secrets to be placed in a cluster, there is an opportunity to integrate K8s with systems storing secrets in order to have them in a secret storage, while K8s can access them securely when needed.

+-Open-source
K8s was created by Google and has been published as open-source. It’s an advantage and disadvantage at the same time as we can face a problem of technical support. In case of an urgent Kubernetes platform issue, there is no opportunity to address it to experts responsible to fix it. So, whoever uses K8s, should understand that for now just the open-source community supports it.

But business needs support for software platforms. So there were the companies who decided to release Kubernetes as a commercial product. One of them is RedHat - a company that provides supported open-source software products to enterprises. They released their own container orchestration platform based on Kubernetes named OpenShift with enterprise lifecycle support.

It is also worth mentioning Rancher that offers the functionality of commercial K8s distribution in an open-source package. It is quite popular now since it allows to easily create clusters on bare metal with Rancher Kubernetes Engine as well as on hosting providers such as AKS and GKE. Rancher is not a redesign of Kubernetes, but rather a platform that makes deployment and use of it easier even in multi-cluster environments.

- High entry threshold
Kubernetes interface is implemented as a number of rather complex Manifests, or descriptions for each abstracted kind. To understand all the peculiarities and to have the ability to configure and debug such systems, one should have a good understanding of microservice architecture and containerization principles, and, therefore, should have a good knowledge of Linux itself as a basis. So if it’s a small or mid-sized project, it would be a tough start for a developer to work with Kubernetes from the scratch without a fair knowledge of the technologies mentioned.

As for the projects, there are cases when startups hope for the fast start and high load of their systems, and eventually, their expectations fall short, which means there were spent extra efforts and resources for implementing microservices and orchestration, and as the result - overhead. This point leads us to the question below.

Is it always Kubernetes?

Since Kubernetes has been on the hype for the last several years, it’s a common practice when a client with a small-sized project requests Kubernetes to be used from the very beginning.

But Kubernetes must be used consciously. Working with Kubernetes should be considered when you have a really large number of microservices, when there are certain requirements for the level of availability of your project/system, when several teams are working on the application, or when infrastructure is needed to automate deployments and tests more efficiently. At this point, it's really worth thinking about Kubernetes.

If you are a small company or if you have a monolithic app which will be hard to split to microservices, Kubernetes is definitely not the best decision.

In the case of a microservice solution, Kubernetes is rather a blessing. Gradually, the application becomes overgrown with logic, and there are more and more microservices. And Docker itself is already becoming insufficient. Plus to everything, the clients probably also want some kind of fault tolerance.

Gradually, the company bumps into the ceiling when they need a fresh and highly productive solution. And here, of course, you need a container orchestrator. K8s is a type of software that manages all microservices, looks after them, repairs, transfers within a cluster, builds networks and routes their traffic, and, in general, is such an entry point to the entire infrastructure of the project.

Summarizing
The IT industry is highly flexible and changeable. The software is more and more complex each year. To keep up to date, businesses need to use the newest tools and solutions.

If we look at this issue in terms of deployment, a couple of years ago using containers in production was considered as an unreliable solution. However, time does not stand still, the industry evaluated and appreciated the prospects of containers. Now the variety of solutions based on containers is more and more convenient and attractive for businesses.

As for the Kubernetes, if implemented and maintained correctly, it offers developers, DevOps engineers, and business owners great benefits: scalability, workload portability, improved app development/deployment efficiencies, effort and cost optimization.

Node.js vs Java: Why Compare?

IntexSoft — Fri, 11 Dec 2020 13:10:49 +0000

Recently, my colleagues and I discussed the popularity of a couple of technologies - in particular, Java and node.js. After a brief Internet surfing session, it turned out that these technologies are used by many information giants to develop and maintain their platforms. Below, I will mention only a few of them.

Companies using Java:

Companies using node.js:

One more interesting fact is that according to indeed.com (06.28.2019) Java developers (30272 jobs) and node.js developers (7401 jobs) are quite in demand.

Why they should be compared

Java is a language, while node.js can be called an ecosystem built on the basis of JS, and, above all, on the basis of Google’s V8 engine.

However, when we talk about Java, we are talking not only about the language but about the Java Virtual Machine (JVM), as well as the entire ecosystem and infrastructure built around this machine. That’s the first reason why we can compare them.

As a result, in both cases, we have the execution environment. In the case of Java, it’s a virtual machine. In the case of node.js, this is the V8 engine that presents on most operating systems, such as Windows, Linux, MacOS, and those lesser known. Developers can write code using the same language, and this will work in more or less the same way on different operating systems due to the fact that there is a runtime environment. The execution environment influences how it interacts with the OS. In addition, they can be compared because they are used to solve a similar range of tasks.

V8 and JVM

When a JS code enters the V8, it is compiled into a byte code due to the just in time(JIT) compilation that is used in the virtual machine, therefore the JS code runs faster.
Byte code is an intermediate, high-level language, so in a JVM, they write not only in Java but also in Scala and Kotlin.

There are prerequisites that in the near future for V8 it will be possible to use not only JS but also TypeScript or others. At the moment, a transpiling of these languages in JS is going on. The perspective for the technology will see that, they will probably be supported out of the box, and everything will work much faster.

Now, when the continuous development of V8 takes place, the emergence of new node.js versions is associated with the appearance of a new version of the V8 engine. They are directly interrelated.

Pros and cons of node.js

Node.js was created by Ryan Dahl in 2009.

The node.js itself includes several key components:

V8 engine;
libuv library, which is responsible for the central part of node - the event loop, which interacts with the OS, as well as for asynchronous I/O;
a set of different JS libraries and the JS language itself.

Now let us turn to the pros and cons of node.js.

Pros:

ease and speed of code writing;
lightness;
simplicity (in comparison with Java);
node package manager (a huge number of libraries that can be installed in one line);
each library falls into a dependency tree and it's all done easily;
constant development (TypeScript is being actively developed, which introduces typing into JS, decorators and is used, for example, for Angular).

Cons:

flexibility and rapid development also causes disadvantages, due to the need to constantly monitor updates, some things are not tested enough;
there was a case where a developer deleted his library from NPM and many applications using it stopped working;

Pros and cons of Java

In contrast, let's consider the main characteristics of Java.

Pros:

speed;
prevalence (Java is taught in universities of many countries);
huge set of libraries.

Cons:

heaviness;
some Java paradigms were created long ago and are already outdated;
JDK is proprietary, so Java is developing slowly.

Recently, JS has begun to overtake Java. Java is also leaving the Android world: it is being replaced by Kotlin, which, although using the JVM, is still a different language.

Conflict of Oracle and Google

Source: giphy.com

Java was created by Sun, which was later acquired by Oracle. For this reason, for many companies, the use of Java causes some issues.

Google had problems when Oracle started a trial with them to use Java on Android. Because of this, Google has very actively adopted Kotlin, which appeared independently.

Java is proprietary. But there is an Oracle virtual machine, as well as an open Java virtual machine (open JVM), which is used in Linux and written in open source. Sometimes there are some incompatibilities, but in recent times these have been fewer.

By the way, Google could not completely abandon Java. In Dalvik, which is used as the core in Android, JVM is utilized. Perhaps they will change this one day, but it will be very difficult since the entire Android ecosystem is built on Java, specifically on an upgraded JVM. And at some point, this was also the cause of the conflict between Oracle and Google, because Oracle prohibits modernizing the JVM. This is the most important part of Java. However, the language itself can be used with no restrictions.

Java vs node.js: performance and resource intensity

First of all, it should be noted that Java performance is much higher than on JS, and, accordingly, node.js.

Performance of node.js and Java

If you run a simple task, like squaring, then in tests the indicators can differ up to 10 times. If you run loops in millions of calculating tasks, Java will almost always exceed node.js.

Plus, the huge difference between Java and node.js is that node is single-threaded, that may be considered its advantage, and its disadvantage on the other hand.

Java can work with threads that are supported at the OS level, and it turns out that a program written in Java makes the most of the OS features. And if you need to write a high-load application that will use a large number of calculations, then Java will definitely work better for this.

The problem is that even a small server written in Java will take up a lot of memory - both on disk and operational.

Node.js is lightweight due to its event-based architecture. It is built to work as a web server and copes very well with servicing lightweight tasks. For example, a simple query like calculating, or writing to a database happens very quickly. And if there are a lot of requests and we want to scale the system into a node, you can use the Nginx or Apache web server. You can have many identical node instances. Then everything will be distributed through load balancing on round-robin. If we run 8 node instances on 16 cores respectively, the OS itself will distribute the instances between the cores. Node does not control this, it will have one thread.

Thread control in Java and node.js

In Java, we can create an application and run 8 threads in it. Due to the fact that there is a closer interaction with the OS, you can distribute the load.

One of the well-known web servers written in Java is tomcat. There you can clearly see that when a user makes a request, additional threads are started. And when the request for a node arrives, the event loop will be processed and sent back, then the next request will appear. And, due to the fact that we are not waiting for the results of the first, it will also be picked up. If the requests are lightweight, all is great. However, when a heavy computation is performed, if there is even one instance, the node stops and a timeout occurs.

Thread control in Java

In node, you can write literally a few lines of code and get a simple web server. Naturally, for a wider functionality, where there will be notifications, authorizations, logging, etc. It is more difficult to implement, but there are frameworks that allow you to solve such issues.

Thread control in node.js

Java has a developed API - concurrency api, which allows you to work with competitive streams. But at the same time, one of the problems as concurrency is that it is a pretty difficult thing which not every developer understands well enough to be able to implement.

Web, REST API is the node’s thing, and sometimes it is used for that. But if we are dealing with complex calculations, it’s still better to use Java.

My Java project

I had an interesting project in Java - a distributed application, where the key task was the processing of large volumes of graphical information for further use in catalogs. When creating a catalog, it is necessary to prepare sets of a large number of images in various resolutions that will be used when creating the catalog. In short, this is an application to automate prepress catalog preparation.

Previously, photographers had to do everything manually. In the beginning, it was necessary to use some small application in order to upload all the images. Next, the specialist who created the catalog had to develop the catalog structure using another application. Then, in another application, they should create a workflow, which scattered the pictures into the structure that was created beforehand. In general, the process was quite difficult. ImageMagick which is available for Linux, Windows, and MacOS was used. We worked with Linux.

For example, a .tiff image with a size of 200-300 MB was loaded into the application; it was necessary to make pictures in various resolutions, cut something out, or make a background.

The first version of the application could not cope with a large load; even a server with a 16-core processor was not enough. We improved the architecture of the application to use several instances at the same time, in order not to radically change the operation of the application. Many instances started up and interacted with each other and each of them processed a piece of the task. It was difficult, but we managed to successfully implement everything in a couple of months. And the system is still working. The process had to deal with competitiveness and various aspects of interaction.

Something from this project could still be done in node, but some things would still have to be done in Java since there were a lot of different calculations. Basically, we could make parts in node that would call certain parts on Java and use its microservice architecture. We could use a mixed version. But this approach does not always work, because a developer specializing in node may not be a Java expert and vice versa. And finding multi-talented developers is much more difficult.

From experience on node.js

There was a project for organizing a large amount of data. Something similar to the project described above. But here we have an uploaded file that contains a large set of information that is subjected to validation through a third-party service (written in Java), several times and according to different rules. It was necessary to process hundreds of gigabytes of information, and node was not intended for this.

Designing the system architecture was especially interesting since the application consisted of several microservices, including third-party ones. When working with a third-party service that performed validation, we used RabbitMQ message broker. We gave the third-party server the necessary information and received a message from RabbitMQ at the end of validation, then the data was processed in sections to avoid an out of memory error.

If the application initially processed a file containing 10,000 entries, it can now process up to a million. We still managed to solve this problem with the help of node.js, although in Java it could have been solved more simply. The client insisted on using node.js since they needed a single infrastructure and architecture with microservices written in JS. Using node to solve the problem was much more difficult, and took more time, but node.js wins due to its scalability. That is why now we can increase the number of workers and process and more and more data. In Java, this would be more complicated.

In fact, any problem can be solved both ways, but as I’ve said before, if there are a lot of calculations, then it is better to use Java, if not then use node.

Results and prospects: can node.js overtake java?

Now there’s a tendency that node.js will often be used as a “wrapper”, and the “stuffing” will be written in other languages. Its shortcomings are already well noted. For example, such a conditional flaw as single-threading has already been fixed. The latest version of node provides the ability to use multiple threads.

Java was originally created as a lightweight solution replacing C++ and now has become heavyweight. It’s like an evolution. Maybe someday there will be something that will replace node.js as well.

http://www.modulecounts.com/ - Development of Java vs node.js

At the moment, according to the number of requests, and to my own feelings, node.js has already overtaken Java. JS is actively developing and it will continue to do so. But, at the moment, there is no potential competitor that could replace Java or node.js.

The problem is that lately Java is developing quite slowly, and node.js is developing at such a rate that it just cannot be replaced in the nearest future.

Which technology do you use in your work? Share your thoughts, interesting projects and questions in the comments.

Selenide Test Automation: Using Selenoid in the Docker Container

IntexSoft — Thu, 19 Nov 2020 15:12:11 +0000

Selenide Test Automation: Using Selenoid in the Docker Container
This article will be useful for young QA specialists, as well as those who are interested in the features and capabilities of such popular testing frameworks as Selenide and Selenoid.

Here we will look into some basic Selenium project. I will describe how to connect Selenium and TestNG to the project and will see a Page Object example with the description of page elements and the methods used.

The next step is the acquaintance with Selenide: we will see into the framework itself, look through its main features and advantages of usage. Will learn how to add Selenide to a testing project, how to work with elements, assertions, and waits available in Selenide.

And finally, I will connect the Selenoid framework to the project to run tests in the Docker container and outside it.

1. Selenium + TestNG. Selenium Maven

We are considering a project build on the Maven builder, so we can find the project structure description in the pom.xml file. To order use Selenium+TestNG, we should add appropriate dependencies to pom.xml file. You can observe them between the dependencies tags below:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"                               
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">    
    <modelVersion>4.0.0</modelVersion>
        <groupId>test</groupId>
        <artifactId>test</artifactId>    
        <version>1.0-SNAPSHOT</version>    
        <dependencies>             
                <dependency>                       
                      <groupId>org.seleniumhq.selenium</groupId>                      
                      <artifactId>selenium-java</artifactId>                     
                      <version>3.141.59</version>                 
                </dependency>           
                <dependency>                      
                       <groupId>org.testing</groupId>                     
                       <artifactId>testing</artifactId>                     
                       <version>6.14.3</version>                     
                       <scope>test</scope>           
                </dependency>   
        </dependencies> 
</project>

Here is the example of the Page Object:

import...

public class SignUpPage {
   private WebDriver driver;

   public SignUpPage(WebDriver driver) { this.driver = driver; }

   private By emailFriend = cssSelector("#register-email");
   private By confirmEmailFriend = cssSelector("#register-confirm-email");
   private By passwordField = cssSelector("#register-password");
   private By displayNameField = cssSelector("#register-displayname");
   private By monthDropDown = cssSelector("#register-dob-month");
   private By dayField = cssSelector("#register-dob-day");
   private By yearField = cssSelector("#register-dob-year");
   private By shareCheckbox = cssSelector("#register-thirdparty");
   private By registerButton = cssSelector("#register-button-email-submit");

   public SignUpPage typeEmail(String email) {

    driver.findElement(emailField).sendKeys(email);
      return this;
  }

  public SignUpPage typeConfirmEmailFriend(String email) {...}

  public SignUpPage typePassword(String Password) {...}

  public SignUpPage typeName(String name) {...}

  public SignUpPage setMonth(String month) {...}

  public SignUpPage typeDay(String day) {...}

  public SignUpPage typeYear(String year) {...}

As we can see, there is a description of variables with locators for the elements of the registration page at the top of the java file. There are methods directly for interacting with elements of our page below the variable section.
Let’s open the tests themselves:

private WebDriver driver;
private SignUpPage page;

@BeforeMethod
public void setUp() {
       System.setProperty("webdriver.gecko.driver", "C:\\Users\\Nikita\\IdeaProjects\\autotests_examples\\drivers\\geckodriver.exe");
       driver = new FirefoxDriver();
       driver.manage().timeouts().impicitlyWait(10, TimeUnit.SECONDS);
       driver.get("https://www.spotify.com/us/signup/");
}

As we can see, in the @BeforeMethod annotation we describe what will have before each method.

@Test
public void typeInvalidYear() {
     page = new SignUpPage(driver);
     page.setMonth("December");
                 .typeDay("20")
                 .typeYear("85")
                 .setShare(true);
      Assert.assertTrue(page.isErrorVisible("Please enter a valid year."));

The @test annotation provides code for test methods.

@AfterMethod
    public void tearDown() {
        driver.quit();
    }

The @AfterMethod annotation contains the code that should be executed after each method.
When running tests using Selenium, the following will happen:

Opening a separate browser window
Following the url
Test code execution
Closing browser session after each test case When running the next test the same things will happen. It should be mentioned that running tests on Selenium is rather a resource-consuming process.

2. Selenide: what, where, and how

So what is Selenide itself? What are its main features and advantages?

In short, Selenide is a wrapper around Selenium WebDriver that makes it quick and easy to use when writing tests. At its core, Selenide is a tool for automating user actions in a browser, focused on the convenience and ease of implementing business logic in autotests in the user’s language, without being distracted by the technical details of working with the “browser driver”. For example, we do not need to focus on working with the waiting for elements in the process of automating testing of dynamic web applications, as well as on the implementation of high-level actions on elements.

Key and main advantages of Selenide:

Concise jQuery-style syntax
Automatic handling of most problems with Ajax, waitings and timeouts
Automatic handling of browser lifecycle
Automatic creation of screenshots

The purpose of Selenide is to focus on business logic of tests and not “waste” mental energy on technical details.

Getting started with the Selenide

To get started with the Selenide we need to add the Selenide dependency to the pom.xml file. Since we no longer need the Selenium dependency, we simply remove it.

<dependency>
            <groupId>com.codeborne</groupId>
            <artifactId>selenide</artifactId>                             
            <version>5.2.8</version>
</dependency>

In order to start using Selenide in our project, we also need to make some imports. Here are the examples of import required classes:

import static com.codeborne.selenide.Selenide.*;
import static com.codeborne.selenide.Selectors.*;
import static com.codeborne.selenide.Condition.*;
import static com.codeborne.selenide.CollectionCondition.*;

For more information on how to connect Selenide using the rest of the project builders, see the Quick start section of Selenide documentation.

Working with elements, assertions, and waits

Let’s move on to the Selenide elements and consider the assertions and waits available in Selenide.

import...

public class SignUpTest {
   private SignUpPage page;

   @BeforeClass
   public static void setUp() {
      baseurl = "https://www.spotify.com";
      browser = "chrome";

   }

We replace the BeforeMethod annotation with the BeforeClass annotation in the test file since we no longer need it. Selenide eliminates the need to write Before and After methods as Selenide takes care of the AfterMethod function. We only have the BeforeClass annotation left to register a pair of properties.

We registered the property baseurl, which is in the configuration class and in the BeforeClass annotation and it will be the base url. Therefore, the driver.get that we used in our Selenium tests is no longer needed. We set the browser on which we will run our tests in the property browser.

We can completely abandon the Selenium driver in our test project, Selenide will take care of all the work, encapsulating it in its classes. We will only have to focus on the logic of the tests themselves.

Let’s proceed to using Selenide on our page:

public SignUpPage open() {

         Selenide.open (relativeOrAbsoluteUrl: "/us/signup/");
         return this;
    }

    public SignUpPage typeEmail(string Email) {

         $(emailFriend).sendKeys(email);
         return this;

   }

When invoking the open method, Selenide itself starts a browser session and opens a web page. Selenide also makes sure the browser is closed at the end. Within Selenide.open we can write either the whole http url path, or we can write a relative url. Since we indicated an absolute path as a baseurl, within the Selenide.open method it’s enough to indicate just “/”.

public SignUpPage typeEmail(String email) {

    $(emailField.sendKeys(email);
    return this;
}


public SignUpPage typeConfirmEmailField(String email) {

    $(confirmEmailField).setValue(email);
    return this;
}

In order to find an element using Selenide, we should indicate $ instead of driver.findElement command used in Selenium. I.e using a one-character method we can find directly the element itself. The search method is accepted as a string, similar to the jQuery JavaScript library by default.

In order to find a list of elements using Selenide, we should indicate $$ characters. Instead of List , we write the ElementsCollection command that is already extended with additional methods.

To work with elements we can use both standard Selenium methods (sendKeys()) and setValue() method or its short version vаl().

As you can see, Selenide methods are more understandable. Method click() remains the same, though Selenide has several click() methods: contextClick() (right mouse button imitation) doubleClick() ( imitation of double click on element ) and so on. Having a certain element found, we can continue the search using other locators.

The difference between Selenide find() method and Selenium driver.findElement(By) is that Selenide find() can immediately receive CSS selectors and operate with the Selenide elements, not the Web elements. Basically, Selenide-elements are a more “smart” alternative to Selenium web elements.

Selenide already contains those methods, which would have to be done through an action class or some other way. Selenide allows writing brief and “nice” methods that are understandable for everybody. Selenide is also rather flexible, and due to that, we can use standard Selenium features.

You can find more information about Selenide methods in the official documentation.

Let’s look into wider and more understandable verification examples provided by Selenide:

page.getError("Please enter a valid year.").shouldBe(Condition.visible);
page.getError("When were you born?").shouldNotBe(Condition.visible);
page.getErrors().shoildHave(CollectionCondition.size(6));
page.getErrorByNumber(3).shouldHave(Condition.text("Please enter your birth month."));

Selenide verification scheme allows us to take any element, find it and use the following assertions for it: should, shouldBe, shouldHave, shouldNot, shouldNotBe and shouldNotHave. Depending on the logic and our needs, we use certain “should-methods”. When we want to check if the element exists, we use should(exist). When we want to check if the element is visible, we use shouldBe(visible) method and so on. In fact, we use only three assertions: should, shouldBe, shouldHave, or opposite to them shouldNot, shouldNotBe, shouldNotHave.

Verifications of elements and element collections on Selenide are carried out with the help of methods (assertions) described above. They play role of explicit waits in Selenide: they wait for a condition for a certain element to be satisfied.

Formulations in Selenide are quite logical and understandable. We can write our methods either using the development environment hints or using our logical assumptions. And of course, we can always take a look at the code for implementing the necessary methods described in the documentation, or we can look through the implementing of the method itself.

Automatic screenshots in tests

For JUnit:

In order to take a screenshot automatically after each failed test, we can make an import and indicate the Rule.

import com.codeborne.selenide.junit.screenShooter;

@Rule
public ScreenShooter makeScreenshotOnFailure = ScreenShooter.failledTests();

But in fact, it’s a rudiment, since Selenide has been taking screenshots automatically when tests fail for quite a while. It’s very convenient for error analyzing. Selenide saves all the screenshot to a build/reports/tests folder by default.

In order to take a screenshot automatically of each test (even succeeded), we use the following command:

@Rule 
public ScreenShooter makeScreenshotOnFailure = ScreenShooter.failedTests().succeededTests();

For TestNG we also make an import:

import com.codeborne.selenide.testng.ScreenShooter;

@Listeners({ ScreenShooter.class})

In order to take screenshots automatically after succeeded test, we invoke the following command before running tests:
ScreenShooter.captureSuccessfulTests = true;
We can also make a screenshot with a single line of code at any moment:

import static com.codeborne.selenide.selenide.screenshot;

screenshot("my_file_name");
Thus, Selenide will create two files: my_file_name.png and my_file_name.html

3. Docker: features and advantages of usage

Let’s proceed to Docker itself and its advantages:

Rapid Deployment. There is no need to set up additional tools, we can run them in containers
Convenient encapsulation of applications
Clean monitoring
Easy scaling
When we talk about Docker, the following things should be clarified:

Container is a running instance that encapsulates required software. It consists of images. And it can easily be deleted and created again within a short period of time.

Container image is the basic element of each container.

Docker Hub is the main public Docker repository provided by Docker Inc. It stores a lot of container images. The service is a source of “official” images made by the Docker team or created in collaboration with developers.

Docker installing

To install Docker for Windows, we open the https://hub.docker.com and download the Docker Desktop app for Windows or MacOS.

To install Docker for Ubuntu Linux, we need the sudo apt install docker.io command.

Then we need to run Docker and configure it to start automatically when the system boots by executing the following commands:

sudo systemctl start docker
sudo systemctl enable docker

4. Selenoid: features and advantages

Selenoid is a server, that launches isolated browsers within Docker containers.

Advantages of Selenoid usage:

Single environment for a parallel launch of automated tests
Isolated environment: Selenoid allows running each browser in a separate container, which enables full isolation of the browser environment
Scalability: Selenoid environment does not affect qualitative and continuous testing
Resources consumption and utilization: Selenoid enables to maintain a high load without additional waste of resources; in addition, all inactive containers are removed at the end of each session. Thus, the level of free memory is always appropriate
Installation: Selenoid requires little time and effort. And in fact it is done with the help of one command
Simultaneous support of multiple browser versions: this option is only available if you use Selenoid; several containers with the appropriate browsers are to be built
Focus: challenges may emerge if multiple browsers are run on the same machine within the Selenium Grid. Due to the OS-specific nature, the focus can contain only one window.
Therefore, windows can compete for it. Selenoid allows running each test in a separate container. Thus, this problem is eliminated
User interface and logs: all available logs are accessed easily in Selenoid. There is also the possibility of integration with the ELK stack for faster collection and analysis of current log files

Selenoid installation

Before installing Selenoid you need:

Make sure you have recent Docker version installed (further we look into the usage of Selenoid together with Docker)
The easiest way to install Selenoid is to download Configuration Manager that is used for automatic installation of Aerokube products. Selenoid is such a product
Rename the downloaded file to cm.exe (for easy interaction)
Run the following commands to start Selenoid:

./cm.exe selenoid start — vnc
./cm.exe selenoid-ui start

The ./cm.exe selenoid start — vnc command will download the latest Selenoid version, browser container images, web driver binaries, generate configuration files and finally start Selenoid.

The ./cm.exe selenoid-ui start command installs and starts Selenoid UI. It is a user interface to track what’s happening during the test execution.

Selenoid runs on standard Selenium 4444 port by default. We
can redefine the port using the — port key.

Stats and sessions in Selenoid UI

Selenoid UI is available at the link: http://localhost:8080/
Here we can see the current quota usage, pending browsers and the queue itself. Selenoid UI gets updates via SSE, so there is no need to renew the browser to see what is going on. It will reconnect automatically after any temporary failure.
If we talk about simultaneous testing on different devices, e.g: we have a cross-platform web app with a real-life chat function, we can simultaneously test the interaction between them, that is obviously comfortable.

Selenoid UI capabilities

Selenoid UI has the following capabilities:

You can choose a browser from available browsers and UI will provide a setup example with the right capabilities. We can see from the screenshot that examples are available for several languages.

With the selection of the browser, it could be launched manually right in the interface. While executing tests, we can connect to vnc port in real-time regime, get access to the browser and even intervene in the process of autotests execution.

Logs and VNC

If you use enableVNC=true capability, you can see a list of the available statistics. VNC allows to see and interact with the browser while the log will reflect all browser actions.

VNC session:

VNC fullscreen mode:

You can also see logs of the docker container for each session even without vnc. It means, if you didn’t use — vnc flag, you’ll see logs only.

We can also view recorded videos of our tests. We can access them by opening http://localhost:4444/video/ or by going to the “Videos” tab in Selenoid UI.
Adding Selenoid to run tests within Docker container
In order to add Selenoid into the @BeforeClass annotation we need to do the following configuration:

Configuration.remote = "http://localhost:4444/wd/hub";
Configuration.browser = "chrome";
Configuration.browserSize = "1920x1080";
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(capabilityName: "enableVNC", value: true);
capabilities.setCapability(capabilityName: "enableVideo", value: true);
Configuration.browserCapabilities = capabilities;

Since now we have the Configuration.browser = “chrome” property, we delete Property baseurl which defined the browser for running our tests:

@BeforeClass 
public static void setUp() {
       Configuration.remote = "http://10.0.75.1:4444/wd/hub";               
       Configuration.browser = "chrome";          
       Configuration.browserSize = "1920x1080";      
       DesiredCapabilities capabilities = new DesiredCapabilities();     
       capabilities.setCapability(capabilityName: "enableVNC", value: true);        
       capabilities.setCapability(capabilityName: "enableVideo", value: true);      
       Configuration.browserCapabilities = capabilities;

Selenoid advanced capabilities

Data storage in RAM: Selenoid stores all temporary files in Tmpfs. It is a temporary file repository that allows storing files in RAM. As we know, access to RAM is performed much faster than to the file system of the hard drive.
Various screen resolution types: we can configure the appropriate screen resolution for a running container on their own by setting the required parameters in the Browser Capabilities.
Video recording of tests: it’s possible to record the video of the tests performed. For instance, the activation in the Google Chrome browser is implemented by setting the parameter true in the Browser Capabilities: java ChromeOptions options = new ChromeOptions (); options.setCapability (“enableVideo”, true);

Using Selenoid without Docker

Selenoid uses containers to run browsers, but there are cases when it’s not possible to run a browser within a container. For example, in Windows we have Internet Explorer, that can not be run inside the container. Selenoid can be used as a lightweight Selenium server replacement to run IE, Firefox or Chrome on Windows. For example to use Selenoid with IE.
To do it we need:

Download latest IEDriverServer archive and unpack it to some directory (C:\ in this example)
Download latest Selenoid binary
Create browsers.json configuration file:

{     
   "internet explorer": {      
      "default": "11",      
      "versions": {         
         "11": {            
            "image": ["C:\\IEDriverServer.exe", "--log-level=DEBUG"]          
            }       
        }    
    } 
}

Start Selenoid: ./selenoid_win_amd64.exe -conf ./browsers.json -disable-docker
Run the tests, using endpoint http://localhost:4444/wd/hub with the following capabilities: browserName = internet explorer version = 11
To start Chrome, just download Chromedriver binary and modify browsers.json accordingly
Selenoid does not process launched driver logs by default. So we need to launch Selenoid with the -capture-driver-logs flag to append driver logs for each session into the main log

Summarizing

Solution on the basis of Selenide + Selenoid in Docker container demonstrates high flexibility for configuration of the runtime environment. The stability of this solution, significant time savings when using it and a number of additional features allow us to optimize the process and ensure high-quality software products in a short time. As a result, it is easy to give preference to the above solutions, since they allow us to perform testing automation tasks quickly and accurately.