DEV Community: Ismail Egilmez

Introducing the New Workflow Dashboard: Off-the-shelf observability for GitHub Actions!

Ismail Egilmez — Wed, 25 Jan 2023 08:27:15 +0000

Consolidated workflow monitoring independent from repositories they belong to.

Introduction

As developers, we understand the importance of streamlined and efficient CI workflows. They allow us to quickly and effectively identify and address any issues impacting our codebase and, ultimately, the performance of our applications. That's why we are thrilled to announce the release of a new feature of Foresight - the Workflow Dashboard.

With our new feature, developers can easily troubleshoot CI workflow errors and focus on performance improvement efforts. The dashboard offers a variety of useful insights, including an aggregated view of your workflows with the number of executions, average duration, and the number of erroneous and successful runs. This allows you to quickly identify and address any issues with workflows that are not being fixed or are running slowly.

What is the Workflow Dashboard

The Workflow Dashboard is designed to help users easily monitor and track the performance of their CI workflows over time. It combines the functionality of Foresight’s Repositories view and Highlights, allowing users to filter workflows by time, branch, and name. This means that users can quickly identify and address any issues with workflows that are not being fixed or are running slowly. You'll be able to see at a glance which workflows are causing problems and take immediate action to resolve them.

The Workflow Dashboard also includes a search feature, allowing users to focus on specific workflows across multiple repositories. This is especially useful for teams that are working on multiple codebases or projects at the same time. The search function allows you to quickly and easily find the specific workflows you're interested in without navigating multiple repositories. You can easily troubleshoot any issues that may be impacting the performance of your CI workflows, saving you time and increasing your productivity.

Why Would you use it?

The benefits of Foresight’s GitHub Actions CI Monitoring feature are numerous. You can achieve faster and more reliable analytics for your CI pipelines, troubleshoot errors, detect anomalies, and achieve higher CI visibility and performance.

Another key benefit of Foresight’s GitHub Actions CI Monitoring is the ability to track your workflow issues daily. You can figure out at what point the failure rate began to rise and take action to address the problem. This means you can quickly identify when an issue begins to occur and take action to fix it as soon as possible.

This feature also explains how pull requests (PR) affect your workflow run duration. With this information, you can identify the major impacts of a PR on your workflows and tests and take action to prevent production regressions early in the CI environments. This means you can catch errors before they make it to production, saving you time and resources in the long run.

Foresight also provides granular details, such as breakdowns of duration and failure rates for individual stages and jobs, allowing you to see which steps are blocking your CI pipeline. Additionally, our feature allows you to view your tests' duration, status, and flakiness in a workflow and compare it to previous runs to identify potential performance regressions. This means you can identify slow and flaky tests and take action to improve their performance.

The workflow dashboard also includes detailed trace charts and logs, making it easy to detect unusual behaviors on your GitHub Actions workflows and begin troubleshooting and optimizing. This means you can gain a deep understanding of how your CI pipeline is behaving and take action to improve it.

Workflow Dashboard for Software Teams

Engineering Leaders

As an engineering manager or leader, it's important to have a clear understanding of the performance of your team's CI workflows. This is crucial to ensure that your team delivers high-quality software on time and within budget. With the new "workflow dashboard" feature of Foresight, you can easily monitor and track the performance of your team's workflows over time.

One of the key benefits of the workflow dashboard is the ability to search by workflow name. This makes it easy to focus on specific workflows across multiple repositories, which is especially useful for teams working on multiple codebases or projects simultaneously. This allows you to quickly and easily find the specific workflows you're interested in without navigating multiple repositories.

Developers

You’d want to ensure that your code is integrated and deployed correctly and that any issues are identified and addressed quickly. With the new "workflow dashboard" feature of Foresight, you can easily monitor and track the performance of your workflows over time.

Additionally, the Workflow Dashboard provides granular details, such as breakdowns of duration and failure rates for individual stages and jobs, and view the duration, status, and flakiness of tests in a workflow. This can help developers identify slow and flaky tests and take action to improve their performance.

DevOps Teams

As a DevOps team, it's important to understand the performance of your organization's CI workflows clearly. This is crucial to identify any issues promptly and ensure that the code is being integrated and deployed correctly. DevOps teams can monitor the performance and health of your organization’s workflows with Foresight’s workflow dashboard consolidated from a single pane of glass.

With the workflow dashboard, DevOps teams can better understand the overall health of their organization's codebase, identify any bottlenecks in the CI pipeline and focus on specific workflows for improvement. This helps teams to improve their efficiency and release better software to their customers.

QA Teams

QA teams need to clearly understand the performance of the CI workflows that impact your testing efforts. With the new "workflow dashboard" feature of Foresight, your team can easily monitor and track the performance of specific workflows over time, including the number of executions, average duration, and the number of successful and erroneous runs. This allows you to quickly identify any issues with workflows that may be impacting your testing efforts and take action to address them.

In addition, the workflow dashboard allows QA teams to view granular details such as breakdowns of duration and failure rates for individual stages and jobs. This helps the QA team identify the blocking steps in the pipeline and thus helps to improve the pipeline.

Conclusion

We understand the importance of streamlined and efficient CI workflows for our developer community, and we are committed to providing the best tools to help you achieve this. With the new Workflow Dashboard, we are taking a big step forward in helping you achieve more efficient and effective CI workflows.

Overall, our new GitHub Actions CI Monitoring feature is a powerful tool that helps developers improve the performance of their CI pipelines and deliver better software to their customers.

We encourage our users to check out the Workflow Dashboard and let us know your thoughts. We always seek feedback and suggestions on improving our products to serve our community better.

You can install Foresight’s GitHub application from the GitHub Marketplace. The workflow dashboard feature comes off the shelf without needing any extra configurations. You can find the quick configuration guide of Foresight from this documentation link.

What Is Cloud-based Software Testing and How Can It Enhance Testing Services?

Ismail Egilmez — Tue, 08 Nov 2022 11:09:03 +0000

Due to the sudden upsurge in the usage of software applications around the globe, enterprises are finding it extremely difficult to meet the time to market demands. Enterprise QA teams that are able to detect errors at the earliest will have more time to work on various other development phases as well as enhance the application quality. With the advent of cloud computing technology, enterprises have leveraged several innovative opportunities in software testing and software deployment. It has opened new possibilities, particularly in the domain of software testing & maintenance.

Cloud-based software testing is a set of procedures, tools, and processes that are leveraged by testers inefficiently and precisely testing software. With the utilization of Cloud service models, enterprises can implement testing as a service, without the need to completely invest in testing labs, tools, or infrastructure. Cloud services deal with not just testing but also everything from cloud security, software development, resource utilization, etc.

Without a proper test, the software will always be vulnerable to threats, errors, security breaches, and many more factors that will significantly affect customer experiences. Therefore, while selecting a test tool, infrastructure, or automation service from the cloud, testers must completely understand its platform compatibility, resource support, flexibility, and service cost. Completely ingraining QA, at multiple phases of cloud adoption and administration, must be an overriding mandate. This means enterprises must carry out robust QA planning and stipulation to guarantee that the Cloud services cover the entire test automation process successfully. Moreover, enterprises must be keenly conscious of the growing levels of regulatory scrutiny of Cloud technology. Clients anticipate seeing proof that enterprises have fully completed and implemented cloud-based software testing strategies for data, flexibility, and application exit.

How Can Cloud Computing Enhance Software Testing

Cloud computing technology has revolutionized the way enterprises utilize IT resources across a software development life cycle. From development and testing to the final deployment, it has enhanced the software life cycle and has significantly reduced time as well as cost. Cloud Computing is defined as a platform for facilitating omnipresent, relevant, budget-friendly, on-demand network access to multiple computing resources at a time. Enterprises can leverage everything from networks and servers to storage and tools without having to buy the entire solution. These resources can be rapidly provisioned from the cloud and implemented with minimum administration effort or service provider interaction.

This makes it all the more imperative for enterprises to leverage cloud services across the different business operations.

Here are some benefits for enterprises that leverage cloud computing technology:

1)On-demand self-service: Computing resources such as server performance time and cloud storage can seamlessly be utilized by the enterprise. Once the enterprise signs up for the required services, there will be no need for interaction with the cloud service provider.

2)Broad network access: The cloud interaction or the communication between devices are all based on the cloud network services. Therefore large enterprises that leverage multiple IoT devices, smart devices, machines, or even laptops and phones can seamlessly interconnect and access devices through cloud networks.

3)Resource pooling: Public computing resources are distributed and accessed by several consumers beyond a single enterprise. This type of resource sharing helps companies leverage processing, storage, memory, data centers, and much more from different geographical locations.

4)Rapid elasticity: Computing resources can automatically be provisioned to enterprises for time being based on the need. The resources are mostly under a pay-as-you-use policy which makes it easier for enterprises to implement them or cancel them when not needed.

Benefits of Cloud-Based Software Testing

The majority of the IT companies and software developers are now migrating their legacy systems to a cloud ecosystem for better test automation services. With cloud-based testing, their applications are scalable, flexible, and easily adaptable. Here are some reasons why enterprises are adopting cloud-based software testing over traditional or manual application testing.

It significantly reduces the expenses and the process cycles by sharing the resources when the testing strategy is performed. This is because cloud-based Testing as a Service (TaaS) enables IT and software developers to initialize practical experimental tests on cloud platforms without the necessity to possess licenses or purchasing the resource. This reduces the expenses of testing and improves sharing of resources and the use of services.
Better testing environment of testing and virtual infrastructures. The flexibility of cloud technology enables enterprises to leverage TaaS from any part of the globe as long as the place has a good internet connection. Also, the cloud provides a better virtual environment for testing and SaaS solutions that support the entire testing life cycle, including development. With these virtual infrastructures, enterprises will not have to spend a lot on real labs or traffic generators but just lease the resources from the cloud.
The Pay per use policy of cloud services is the most notable factor for enterprises. As opposed to traditional software testing, in cloud-based software testing, enterprises can choose the resource, tools, or technologies for just the time they need. They will have to only pay for the service based on the utilization time and can stop leveraging cloud services once the testing is complete.

Conclusion

Cloud computing and cloud-based software testing open up new possibilities for QA teams and software development companies. Especially during the current COVID-19 crisis, when there are huge restrictions on human interactions, manual testing will face severe challenges. With the adoption of Cloud-based software testing, enterprises can leverage cloud solutions from any part of the globe at any given time. Even the post-covid era will see a rise in technological innovations, where more applications will have modern advancements such as Artificial Intelligence and Machine Learning integrated into their architecture. To efficiently test such applications and ensure zero errors, enterprises must invest in cloud computing and test automation services.

Bonus Section:

Observability for Cloud-based Software Testing

The shift-left paradigm shows that observability has to be implemented early in the software development lifecycle because it is incontrovertible that any bug in production is more costly than the biggest pre-production failure.

Testing is a must to deliver reliable software to your end-users. Since modern applications are architectured in a distributed way, hosted in the cloud or on-prem, it is difficult to test them and even more difficult to troubleshoot when tests fail.

By being able to gather data from the app in real-time, developers and QA engineers gain 100% code-level observability and faster resolution of test failures. Foresight replaces the iterative, non-agile process required today for debugging failed builds in CI pipelines. Foresight is platform-agnostic, working on-prem, in the cloud, on containers, and on serverless code.

Never miss critical changes in your GitHub Actions workflows

Ismail Egilmez — Thu, 13 Oct 2022 07:01:06 +0000

Summary

Delivering application code at a high pace is possible using GitHub Actions. It lets you automate and manage CICD workflows directly from your GitHub repositories. By using GitHub Actions you can build, test, and deliver code automatically, fast, and safely. You can use any webhook to trigger events such as automatically building and testing code for each pull request.

In other words, you can create your CI pipelines with ease. Theoretically, everything looks fine but when it comes to practice “everything fails, all the time” as AWS CTO, Werner Vogels says.

Foresight is the missing observability piece of GitHub Actions! Foresight provides CI and test observability into your GitHub Actions workflows, helping you maintain their health and performance. In this post, we will cover the need for getting notified about the CI workflow or test failures and how Foresight helps.

What is the challenge?

Missing critical changes that may end up with production bugs is a huge FOMO. Nobody sits all day and monitors their software. But when something goes wrong, missing it can cause serious problems. Developers, DevOps, SREs, testers, QAs, and engineering leaders often face this challenge.

Pinpointing the root cause of errors in your CI workflows and tests as they occur might be vital on some occasions. For example when introducing a new feature to production, it might not have been well tested enough. Or you can easily introduce a bug into production because a flaky test passed and your build ended with success.

Some of the critical challenges that you can overcome by using proactive alerting such as Foresight “rules” are;

Affected Users. Not being aware of errors, latencies or critical changes in your CI workflows or tests may let you introduce bugs to your production environment. Your end users can get badly affected due to this reason.
Cost. Missing critical errors or latencies may cause high costs.
Visibility. Existing APM or error tracking tools lack a holistic view of your test and CI workflows’ health and performance.
Speed. Increased time-to-detect due to a lack of monitoring and observability across CI workflows and tests.
Restoration. Higher mean-time-to-recover because of lacking CI observability and poor notification.
Collaboration issues. Disparate tooling across the organization complicates cross-team collaboration.

Below is a solid use case that happens in an engineering team. In this use case; build times increase and developers don’t know the reason why. It starts impacting the developer productivity and their engineering manager finds a solution.

Anatomy of a rule

Rules lets you set up robust and customizable policies for your workflows and tests. You can receive notifications for fluctuations in key performance metrics as your workflows run.

The below gif showcases how to create a rule in Foresight for your workflows and tests.

Predefined rule templates

We’ve made some rule templates for you to easily set your rules for your workflows within a couple of clicks. We got inspired by our customers’ needs and requirements while creating these rule templates. Below are some examples of the templates we created for you.

Increase in the duration of a workflow compared to the previous week.

Whenever the average duration of a workflow increases from 4mins to 5mins, you will get a notification.

Increase in the failure rate of a workflow compared to the previous week.

When the failure rate of a workflow increases from 10% to 30%, you will get a notification

Workflow failure rate exceeds the limit set

When a failure rate of a workflow exceeds 10%, you will get a notification.

Notification Channels

Foresight rules are a powerful tool for ensuring your CI workflows’ integrity. Notification alerts let you filter on a number of different aspects of your CI workflows’ and tests’ behavior, tying multiple complex objects together behind a unifying rules interface that is both easy to use and high-powered.

You can connect your Foresight rules notifications with incident management tools like PagerDuty and Opsgenie. You can choose directly receive notifications to your email or Slack. Or else, you can connect them anywhere through webhook. With configurable rules, you can rely upon Foresight alerts to provide you with up-to-date information when things go wrong.

How to make your CI pipeline shine

With Foresight, you won’t miss any critical information such as a change, a failure, a latency, etc. from now on. You can receive notifications for fluctuations in your key CI performance metrics.

You can set your rules to get notified about critical changes, and flow them through your Slack, email, PagerDuty, OpsGenie, and more.

You can quickly remediate failing builds and performance regressions, allowing your teams to focus on developing new features. Just set up your alert rule to be notified when a test is longer than usual and optimize slow tests to speed up your pipelines.

Here’s what you can achieve by having an observable CI pipeline that notifies you correctly:

Reliable Products. By being able to pinpoint any deficiencies or regressions as they occur, engineering teams can deliver more robust and reliable products.
Faster delivery. Automating notifications over CI and test failures lead to faster test cycles, which accelerates deployment frequency.
MTTR. Resolving deficiencies quickly before going to production drastically helps to decrease the mean time to resolution.
Productivity. Developer productivity is boosted when their strings are not attached. Developers start spending more time on innovation and improving the customer experience and less time putting out fires.
Easier Audits. You can log your notifications and actions taken afterward, enabling post-incident review.
Executive Trust and Confidence. Because incidents are resolved quickly with little customer impact, executives no longer lose sleep at night and trust that CICD pipelines will stay up and available.

To get started, all you have to do is to install Foresight’s GitHub application and signup for Foresight to get your account. It is FREE for open-source projects and has a very generous free plan for small and medium sized organizations.

TL;DR

Foresight’s “rules” feature brings the ability to stay on top of any changes in your CI pipelines. Missing critical changes that may end up with production bugs is a huge FOMO. To overcome this, we came up with “rules” that make it possible to define potential scenarios and automated actions when prerequisites are met.

To be more precise, you can say “Send an alert notification with a high severity to my PagerDuty when my xyz CI pipeline’s failure rate exceeds 10%”.

Or you can make comparisons such as “Send a notification to my Slack when there’s an increase in the average duration of my xyz workflow compared to the previous week”.

Go ahead and take action to improve your developer productivity. If you realize your current setup isn’t providing enough control over your CI metrics, try using Foresight to proactively monitor your CI workflows and tests.

Up on the horizon, we’ll be coming up with much more advanced querying and alerting capabilities of Foresight “rules”. We’ll be happy to listen to your ideas and needs to facilitate success in your CI workflows using Foresight.

For more information about configuring our integration, refer to our docs. Please submit your requests to support@runforesight.com or come to our Discord community and let’s chat. You can sign up for Foresight or see the live demo to see Foresight in action.

5 Best Logging Solutions for Java

Ismail Egilmez — Wed, 05 Oct 2022 13:21:52 +0000

What is Logging in Java

Logging is the process of recording the application insights as a readable log file. It is an important requirement for any application since it helps identify issues and fix bugs in applications while improving performance.

Logging frameworks are similar to the daily diary of a web application. They keep track of activities, errors, and sensitive details of internal processes. Therefore, picking the most secure and reliable option is crucial when selecting a logging framework.

In this article, I will discuss the best logging frameworks optimized for Java, focusing on their characteristics to help you choose the best.

1. Log4J and Log4J 2 - A well-known logging solution for Java

Log4J(https://logging.apache.org/log4j/2.x/) is a Java-based logging framework. It is a part of Apache Logging Services. It was also the most popular and widely used Java logging solution until the exposure of its Log4Shell vulnerability last year.

Log4J 2 is the actively maintained version of Log4J, and it has some amazing features for developers.

API separation for forward compatibility - The isolated Log4J API from the implementation is a significant feature. It guarantees the forward compatibility of web applications' code. And also, it creates loosely coupled logging framework integration.
Automatic reloading of configuration - During the application runtime, Log4J2 supports reloading the framework if there are any configuration changes without affecting the application and its previous logs.
Asynchronous logging for better performance - Many developers unintentionally make their web applications slow by not focusing on the performance of the logging framework. However, Log4J2 has asynchronous logging, which has significantly large throughput. especially when the program executes in a multi-threaded environment.

However, async logging is not enabled by default. Hence, it is better to check all the pros and cons beforehand through the official Apache documentation if you wish to enable async logging.

As a free and open-source Java logging framework, Log4J 2 is good for web applications that do not deal with sensitive data. On the other hand, there can be security and reliability loopholes due to the openness and majority of users of the framework. So you must constantly check for security updates and patches.

2. Logback - A free and open source modularized logging framework

Logback(https://logback.qos.ch/) is another non-commercial Java logging framework. It labels itself as a successor to the previously discussed Log4j framework.

Logback has three main modules:

logback-core - The base module which is essential for operating the other two modules.
logback-classic - This module implements SLF4J API allowing developers to easily integrate with other logging frameworks if required.
logback-access - This allows communication with Java Servlet containers (e.g., Tomcat and Jetty) so that its framework can provide HTTP access logs.

In addition, Logback has a simple architectural design with some exciting features:

Automatic log archive cleanup: You can configure Logback to delete the archives using the maxHistoryproperty of TimeBasedRollingPolicy or SizeAndTimeBasedFNATP.
Automatic log archive compression: Logback can automatically compress archived logs in an asynchronous manner. So there will be no significant performance impact on your application during the compression process as well.
Easy to implement: This library has a well-modularized design and a more familiar implementation of the famous Log4J framework. It is developer-friendly and quick to set up.

Suppose you are a Java developer looking for a free logging framework but do not want to risk using Log4J or Log4J2. In that case, I recommend you to have a look at Logback.

3. Tinylog - A lightweight logging framework

Tinylog(https://tinylog.org/v2/) is a lightweight logging framework for Java and Andriod applications. It is an open-source framework and can be used with any JVM language like Kotlin and Scala.

Although it is relatively new and lightweight, it brings some unique features:

Static logger class: This implementation allows you to use the logger directly instead of creating logger instances everywhere you need to create logs.
Fast output of log records: The benchmarks prove that the tinylog is well optimized for writing log outputs to files.
Lightweight: Most significant feature of tinylog is the small library size of approx. 178KB only.

If you are looking for a small-size free and open-source logging framework, Tinylog is one of the best options you have.

4. Solarwinds Loggly - A commercial logging toolkit with HTTP/HTTPS log management

Using text-based logging frameworks comes in for practical difficulties like analyzing logs, time-consuming log search, and third-party integrations.

Loggly by Solarwinds(https://www.solarwinds.com/loggly) resolves these issues by providing a log management toolkit and a Java extension that can record logs. It is a commercial product and offers 4 different packages, including a free trial.

Lite - Free version
Standard - Starting at 79 USD per month. This package is annually billed and suitable for small-scale businesses.
Pro - This package, starting at 159 USD per month, is annually billed and suitable for growing businesses.
Enterprise -This package, which starts at 279 USD per month, is annually billed. It includes all the features.

Loggly has user-friendly log management options like:

Centralized Log Monitoring Tool
Automated Log Summaries
Smart Querying of Logs with Filters

However if you are running on a tight budget, using Loggly can be tricky due to its expensive pricing model.

5. Sidekick - An alternative to all logging solutions

As has been discussed, there are many Java logging frameworks with different features. But, there is a major limitation with all of them. They do not allow you to add logs into the built-in dependencies of your web application.

Sidekick(https://www.runsidekick.com) resolves this issue by allowing developers to put logpoints in dependencies. Apart from the simplicity, the below items explain why Sidekick is an alternative to the other logging solutions:

Dynamic logging - This feature allows developers to add and remove logs on demand. You can add logs to your running applications without stopping or redeploying it. This feature not only solves the issues of traditional logging but also does not bring any overhead or security issues.
Ingest collected logs anywhere - Sidekick makes it easier to add new logs to multiple services without rebooting & redeploying and then you can start sending your logs to anywhere you wish. Here are some recipes telling about how you can send your logs to your Loki or Elastic instances.
Integrations with IDEs - This feature is one of my favorites. It allows developers to debug applications within the IDE itself with the Sidekick plugin. At the moment, this integration works well with IntelliJ IDEA, VSCode, and PyCharm.

The below process shows how Sidekick works in a nutshell.

If you are looking for a secure and reliable Java logging framework, Sidekick is the go-to option. As you can use this for free, I highly recommend trying it out by yourself. You can get started from the official documentation below.

Official Sidekick Documentation:

https://docs.runsidekick.com/

Final Thoughts

Selecting a reliable and secure logging framework for Java applications is more complicated than it sounds. Especially with the identified Log4J vulnerability last year, many developers and product owners are highly picky about logging frameworks.

However, in this article, I have presented 5 latest Java logging solutions with their features, including my thoughts about each of them. Based on my hands-on experiences, the following comparison summarizes each of the above-discussed logging solutions.

I hope this article will help you explore the aforementioned frameworks further and pick the best one for your needs. Thank you for reading!

Observability for CI workflows on GitHub Enterprise

Ismail Egilmez — Mon, 26 Sep 2022 13:15:54 +0000

Foresight Plays Well with GitHub Enterprise

Automatic application deployment and delivery are essential to the success of software development projects today. This allows applications to be quickly deployed to production environments and minimizes the risk of issues that may arise when applications are deployed manually. However, the building and testing steps that need to be executed in the software pipelines are increasingly complicated.

Maintaining the pipeline and debugging the issues that caused it to fail can be complex, sometimes taking days to resolve everything—especially if you have to scroll through pages in the GitHub Workflow logs.

Foresight’s system monitoring ensures that you won’t have to search through an ocean of GitHub logs. Foresight’s highlight dashboard allows for quick troubleshooting by indicating which workflow has a problem. Moreover, Foresight process traces allow you to easily see which step in the workflow is problematic.

In this article, we’ll look at how Foresight can help you quickly fix problematic workflows, especially when integrating Foresight into your existing GitHub Enterprise environment.

The best way to learn about Foresight’s features, easy integration with GitHub Enterprise, and the powerful capabilities you get when integrating them is to implement GitHub workflows for a real-world application.

A Demo Application

As an example, we will implement a blog management application using Node.js with Express as the backend framework and MongoDB Atlas as the cloud database. The application will have CRUD (create, read, update, delete) APIs for managing the application’s users and blogs.

First, go to Mongo Atlas to create a new database that has three collections as: posts, revoked tokens, and users.

Figure 1: Mongo Atlas database creation

Then, cloning the application to your local machine, all you need to do is change the database configuration string in the config/db.js file.

 `module.exports = {
    MONGO_CONNECT_URL:"mongodb+srv://donaldle:cuong1990@cluster0.nolan.mongodb.net/myFirstDatabase?retryWrites=true&w=majority"
};`

Next, bring up the application locally and play with its API by running the following command:

 `node index.js`

Use Postman to create a new user with the API.

Figure 2: Creating a new user using the application API

Or create a new blog post:

Figure 3: Creating a new blog using the application API

Our blog management application is now working as expected. The next step is writing the tests.

Writing the Test

In order to make sure the existing functionalities of the application still work after adding new features, I added some tests for the application.

The tests are located in the tests directory. There are tests for authentication, user management, and blog management using Jest as a test runner as well as supertest to call HTTP requests. Below is an example of a test for an authentication API.

 `authenticate_user.test.js

const request = require("supertest")
const baseURL = "http://localhost:3000/v1"

describe("Authenticate user", () => {
    const authUser={
            email:"donald.le@iamondemand.com",
            password:"tatiana"
    }
    it("should return 200", async () => {
        const response = await request(baseURL).post("/auth").send(authUser);
        expect(response.statusCode).toBe(200);
    });
  });`

Here, I defined the request body to send for an authentication API, which includes the user’s email and password information. After calling the API, I expect the status code that returns from the API to equal 200.

Now we will run all the tests to see whether any issues arise.

 `npx jest tests`

Figure 4: Tests are run with pass results

As shown in Figure 4, it passed all the tests. Now we will move to creating GitHub Workflows for the application.

Create Github Workflow

GitHub Workflows help to automatically build the application whenever a change is applied to the GitHub repository. In addition, the API tests are executed so I will be notified if something goes wrong. It is possible to create workflow to build the application and execute the test in the GitHub agent environment for:

User management
Blog management
Authentication

GitHub workflows are defined in the .github folder in the Git repository. For example, the blog management workflow is written as follows:

 `name: Blog Management Workflow
on: [push]
jobs:
 build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Use Node.js
        uses: actions/setup-node@v3
        with:
        node-version: '16.x'
    - run: npm ci
    - run: node index.js &
    - run: npx jest --config ./jest.config.json --collectCoverage --coverageDirectory="./coverage" tests/blog_management/`

Here we have steps to check the code to the GitHub agent, install application dependencies, build the application, and finally execute the tests for the application. Whenever a code to the GitHub repository is updated, the workflow is automatically triggered.

‍

Figure 6: Workflow results in GitHub Actions

After successfully applying the GitHub workflows for the application, the next step is adding a new feature to the application.

Applying API Cache for the Application

In order to boost the performance of the blog management APIs, especially for the APIs that are used for retrieving blog content, we can add the API caching mechanism using the express-api-cache library.

Navigate to the app/route/v1/post.js, update the code for getting the blog by its ID, and get all blogs, as shown below:

 `const router = require('express').Router();
const PostController = require(APP_CONTROLLER_PATH + 'post');
let postController = new PostController();
var cacheService = require("express-api-cache");
var cache = cacheService.cache;

router.get('/', cache("10 minutes"),postController.getAll);
router.get('/:id', cache("10 minutes"),postController.get);
router.post('/', postController.create);
router.delete('/:id', postController.remove);
router.put('/:id', postController.update);

module.exports = router;`

The API for getting all blogs and for getting one blog by its ID is now cached for 10 minutes. After that time, the existing cache will be invalidated and the new cache will be applied.

Pushing the new code to the GitHub repository will cause the blog management workflow to fail.

Figure 7: Recent workflow runs are failed for blog management

Why did the blog management workflow fail after applying the caching mechanism for the APIs? The next step is troubleshooting this problem.

Integrate GitHub Enterprise with Foresight

Foresight is ideal for monitoring workflows and troubleshooting tests in the application. You can easily integrate Foresight with GitHub Enterprise with just a few steps.

First, search for the Foresight application in the GitHub MarketPlace apps and the Foresight app for your GitHub Enterprise.

Figure 8: Foresight application in GitHub Apps Marketplace

Then add permissions in order to give Foresight the capabilities to access your repositories in GitHub.

Figure 9: Add permissions for Foresight to access GitHub repositories

The final steps are adding foresight-workflow-kit-action and foresight-test-kit-action to the existing workflows along with the Foresight API key.

 `name: Blog Management Workflow
on: [push]
jobs:
 build:
    runs-on: ubuntu-latest
    steps:
    - name: Collect Workflow Telemetry
        uses: runforesight/foresight-workflow-kit-action@v1
        if: success() || failure()
        with:
        api_key: b983b8fb-e108-483a-bcd7-fdabdb16944a
    - uses: actions/checkout@v3
    - name: Use Node.js
        uses: actions/setup-node@v3
        with:
        node-version: '16.x'
    - run: npm ci
    - run: node index.js &
    - run: npx jest --config ./jest.config.json --collectCoverage --coverageDirectory="./coverage" tests/blog_management/
    - run: ls
    - name: Foresight test kit
        if: success() || failure()
        uses: runforesight/foresight-test-kit-action@v1
        with:
        api_key: b983b8fb-e108-483a-bcd7-fdabdb16944a
        test_format: JUNIT
        test_framework: JEST
        test_path: junit.xml
        coverage_format: COBERTURA/XML
        coverage_path: ./coverage/cobertura-coverage.xml`

We should now be able to capture the workflow and test results from Foresight.

Figure 10: Successfully integrate Foresight with GitHub Enterprise

In the next section, we’ll look at how to debug the failed blog management workflow.

Debugging the Failed Test with Foresight

Looking at the Foresight highlights, you can see that all the workflow failures are from the blog management workflow.

Figure 11: Foresight workflow highlights

Scrolling down a little bit, you can see the overall status of the workflows.

Figure 12: Summary of workflows in Foresight

Let’s check the details of this problematic workflow for more information.

Figure 13: Find out the step that cause workflow failed

The step that caused the workflow to fail is the test execution.

Clicking on the “Tests” column, we see that 10 lines of new codes were added although I did not add any tests to it.

Figure 14: Check out the change impact analysis of the git repository

When we click on “See changes” we see that there are 4 files that were recently updated, including the app/route/v/post.js file.

Figure 15: Files that have been modified

By clicking on the app/route/v1/post.js file, you can see the line of code that was updated. In the post.js file, I updated 4 lines to apply a caching mechanism for the APIs to get blogs.

Figure 16: Details of the app/route/v1/posts.js file that has been modified

Next, let’s check the processes column in the workflow detail. You can see the process details for the test execution. The process file name in this step is:

 `home/runner/work/foresight-lc-m/foresight-lc-m/node_modules/.bin/jest`

And its process argument is:

 `"/usr/bin/env","node","/home/runner/work/foresight-lc-m/foresight-lc-m/node_modules/.bin/jest","--config","./jest.config.json","--collectCoverage","--coverageDirectory=./coverage","tests/blog_management/"‍`

Figure 17: Processes details captured by Foresight

Navigating to the “metrics” tab, you can see the metrics for CPU, memory, network, and disk.

Figure 18: CPU metrics for the current workflow capture

Figure 19: Memory metrics for the current workflow capture

Figure 20: Memory metrics for the current workflow capture

Figure 21: Disk metrics for the current workflow capture

Comparing the current network metrics to the previous network metrics of this workflow, you can see the difference in network IORx Mb value. The current network metric is only 7 MB, whereas in the previous network metrics the value was 9 MB.

Figure 22: Network metrics for the previous workflow capture

This makes sense because I already applied the caching mechanism for the APIs, so the API for getting blog content does not directly call the MongoDB to get the blog value; it only calls the caching content.

Let’s go to the test runs to see the details for every step in the test.

For the blog management workflow, we have two test suites. The failed one is “Update existing blog.”

Figure 23: Determining the problematic test suite

Click on “Update existing blog” to see the details.

Figure 24: Determining why the test failed

You can see that the test failed, because the expected value is “A new testing blog afterUpdateBlog,” whereas the received value is “A new testing blog beforeUpdateBlog.”

Next, check the test file for more details.

 ``beforeAll(async () => {
    // Get user token
    const response = await request(baseURL).post("/auth").send(authUser);
    expect(response.statusCode).toBe(200);
    userToken=response.body.data.token
    const responsePut = await request(baseURL).put(`/posts/${blogId}`).set("Authorization","JWT " + userToken).send(beforeUpdateBlog);
    expect(responsePut.statusCode).toBe(200);
    })
    it("should return 200", async () => {
    const responseGetBefore = await request(baseURL).get(`/posts/${blogId}`).set("Authorization","JWT "+userToken);
    expect(responseGetBefore.statusCode).toBe(200);
    expect(responseGetBefore.body.data.title).toBe(beforeUpdateBlog.title);

    const response = await request(baseURL).put(`/posts/${blogId}`).set("Authorization","JWT "+userToken).send(afterUpdateBlog);
    expect(response.statusCode).toBe(200);
    expect(response.body.data.title).toBe(afterUpdateBlog.title);

    const responseGet = await request(baseURL).get(`/posts/${blogId}`).set("Authorization","JWT "+userToken);
    expect(responseGet.statusCode).toBe(200);
    expect(responseGet.body.data.title).toBe(afterUpdateBlog.title);
    });``

In the test for updating an existing blog, I have to change the step for the blog content to the initial value as “A new testing blog beforeUpdateBlog,” and then call to get the API blog to confirm the blog has been updated. Finally, I update the blog content to “A new testing blog afterUpdateBlog,” then call the get API blog to confirm.

The first time I call the get-API blog to confirm, the request is made to the MongoDB database. Its value will be cached because I implemented the caching mechanism. The second time when I call the get-API blog to confirm, the request is not made to the MongoDB database, but calls the cached value from the previous step. The cached value from the previous step is “A new testing blog beforeUpdateBlog,” even though I expected it to be “A new testing blog afterUpdateBlog.”

This is why the test for “Update existing blog” and the blog management workflow both failed.

Conclusion

Implementing workflows for automatically building and testing your web application is difficult; maintaining them over time is even harder. Without the right support tools, you could end up spending most of your time debugging failed workflows rather than implementing new features for your application that actually fulfill the business requirements.

Foresight does all the heavy lifting for debugging, tasks are already done. By simply looking at the Foresight highlights and process traces, you’ll know exactly which part of the workflow has problems so you can address them. This is even easier if you’re already using GitHub Enterprise: Foresight, now a partner with GitHub, provides the support you need when integrating with GitHub Enterprise.

Save yourself the frustration of scrolling repeatedly through GitHub logs for debugging tasks! Check out Foresight.

Enhancing Software Change Impact Analysis

Ismail Egilmez — Mon, 12 Sep 2022 14:19:25 +0000

A comprehensive test suite is part of modern software development best practices. Unit tests, integration tests, end-to-end tests, and many others make sure your system keeps working when you need to change the implementation.

But over the lifetime of a software project, you can end up with hundreds of such tests, and every test you add can gradually slow down your CI/CD pipeline. You have high standards for software quality, but with all these tests, your development velocity goes down the drain. How can you get both: quick changes and high quality?

How Does Change Impact Analysis Help?

Change Impact Analysis is one solution to this problem. The goal of it is to minimize the number of tests required to run for a specific change. When you have a massive suite of tests, change impact analysis helps you organize and track which test impacts which source files. Later, when you change one or more of these source files, you can take the change impact analysis list and check which parts of your test suite are impacted by the changes you made.

With change impact analysis, you only run the tests that matter, save time in your CI/CD pipeline, and get results quickly.

Implementing Change Impact Analysis for Node.js

Let’s find out how we can set up change impact analysis for Node.js. For this project, we’ve created an example repository on GitHub that you can clone locally and use as a reference.

This example project is an API that uses the Express framework. It consists of three routes, each having its file and four test files—one for every route and one with tests for two routes.

The unit tests are implemented with the Jest testing framework, which supports code coverage output. This output allows us to check which test file affects which source file.

src/

index.js

route-a.js

route-b.js

route-c.js

tests/

route-a.js

route-b.js

route-c.js

route-a-c.js

Running all tests

If you run the test:all you will execute all tests in the __tests__ directory.

$ npm run test:all

This is the default behavior, and in our example, this is no problem. But, over time, it could grow slower. That’s why we have to use change impact analysis to find out which test file is concerned with which source file.

Analyzing Change Impact

To analyze change impact, we have to complete the following steps:

Check which test files exist
Run each test file on its own
Check the resulting code coverage of each test run
Write down which source files the test touched

To find out what source files were touched, you need to configure Jest to output code coverage data; in this project, that’s done inside the package.json file.

This is the code for the Jest configuration:

 "jest": {
  "collectCoverage": true,
  "coverageDirectory": "coverage",
  "coverageReporters": [
    "json"
  ]
},

In the scripts directory is a init-tia.js file, which implements these steps. Let’s look at the important parts of this file:

 const testFileNames =
  fs.readdirSync("./__tests__")
const testImpact = {}
for (const testFileName of testFileNames) {
  await jest.run(testFileName)
  const absoluteFilePaths = Object.keys(
    JSON.parse(
      fs.readFileSync(
        "./coverage/coverage-final.json"
      )
    )
  )
  testImpact[testFileName] =
    absoluteFilePaths.map((f) =>
      f.replace(process.cwd(), ".")
    )
}
fs.writeFileSync(
  "./scripts/tia.json",
  JSON.stringify(testImpact, null, 2)
)

The script gets all of the test files and loops through them. Each loop iteration executes Jest with just one test file and reads the disk’s coverage report. Next, the script adds an entry to the change impact analysis list that contains the test file name and the corresponding source file names. Before writing the source file

names, it transforms the file’s absolute path into a relative one.

Finally, the script writes the change impact analysis list as JSON to disk, so a future execution of the test suite can use it as a filter. The list looks like this:

{
  "route-a-c.js": [
    "./src/route-a.js",
    "./src/route-c.js"
  ],
  "route-a.js": [
    "./src/route-a.js"
  ],
  "route-b.js": [
    "./src/route-b.js"
  ],
  "route-c.js": [
    "./src/route-c.js"
  ]
}

We can see here that most tests only affect one source file, but theroute-a-c.js test affects two source files. As a result, the tia.json file must be checked into the source control and updated every time the test suite changes.

Running Only Impacted Tests

Now, we can run only essential tests. But let’s look at the run-tia.js script before we run it.

 const requiredTestFiles = []
  const allChangedFiles = JSON.parse(process.argv[2])
  const { FORESIGHT_JEST_ARGUMENTS } = process.env
  if (FORESIGHT_JEST_ARGUMENTS) {
    const foresightArgs =
      FORESIGHT_JEST_ARGUMENTS.split(" ")
    requiredTestFiles.push(...foresightArgs)
  }
  const changedSourceFiles = allChangedFiles
    .filter((file) => /src\/.*.js/gi.test(file))
    .map((file) => "./" + file)
  for (const sourceFile of changedSourceFiles)
    for (const testFile of Object.keys(tia))
      if (tia[testFile].includes(sourceFile))
        requiredTestFiles.push(testFile)
  if (requiredTestFiles.length < 1)
    return console.log(
      "No tests cover the changed files. Aborted."
    )
await jest.run(requiredTestFiles)

The idea here is that the script will get a list of changed files from the CI/CD pipeline as an argument. It will then check which tests cover the files and pass the tests to the Jest library. This way, only tests that are related to the changes will be executed.

If we look at the GitHub Action definition, we can see where the changed files come from.

 steps:
  - id: files
    uses: jitterbit/get-changed-files@v1
    with:
      format: json
  - uses: actions/checkout@v2
  - name: Use Node.js $0
    uses: actions/setup-node@v2
    with:
      node-version: $0
  - name: Install dependencies
    run: npm ci
  - name: Run tests
    run: npm run test:tia -- '$'
    if: $

The files step will gather all files changed in the push or PR that triggered the action. The Run tests step will only execute if a file in the src directory has changed and passed a JSON array of these to our test:tia script.

To run the tests, make a change inside a file in the src directory, commit it, and push it to GitHub. GitHub Actions will start automatically right after the push.

Getting More Insight with Foresight

It’s nice to filter out tests that aren’t needed, but we also want to make the most of the tests we keep and continue to execute—and there’s no better tool for optimizing your tests than Foresight. It can be integrated simply by installing Foresight's GitHub app on the GitHub Marketplace and changing the “Run tests” step in the run-test-tia.yml file.

First, create a Foresight account and connect your pipeline by simply installing the GitHub app. After installing the app successfully, you will see the project creation and repositories screen. You should name your first project and select the repositories you want to monitor.

In order to gain insights about our tests such as grouping tests, test suites along with their logs, screenshots, and more to understand why even the most complex integration test failed; we need to update our YAML file with Foresight's report uploader step and troubleshoot our test failures easily.

We can change the GitHub action for our tests to use the Foresight integration.

steps:
      - id: files
        uses: jitterbit/get-changed-files@v1
        with:
          format: json
      - uses: actions/checkout@v2
      - name: Use Node.js $0
        uses: actions/setup-node@v2
        with:
          node-version: $0
      - name: Install dependencies
        run: npm ci
    - name: Foresight Test Report Uploader
          if: always()
      uses: actions/upload-artifact@v2
      with:
        name: test-result-jest
        path: ./target
     - name: Run TIA tests
        run: "npm run test:tia -- '$'"
        if: $

That’s it. Now we can change a file in the src directory again, commit it, and push it to GitHub to see if everything is working.

It will take a few minutes to get the results in Foresight, but after that push, you can go to the Foresight web console, open your project, and wait for the results.

Foresight provides full visibility and deep insights into the health and performance of your tests and CI/CD pipelines. You can assess the risk of changes, resolve bottlenecks, reduce build times, and deliver high-quality software at speed.

Conclusion

Big test suites give us peace of mind when modifying our code, but they also slow down the delivery of new releases—the bigger the test suite, the slower the pipeline. With change impact analysis, we have a solution to that problem. If we only run tests related to our changes, we can save time while still checking the crucial parts of our code. Sign up for Foresight to improve your test runtimes and gain insight into your tests.

The Known and Unknown Costs of CI Testing

Ismail Egilmez — Mon, 12 Sep 2022 13:59:18 +0000

The purpose of software testing is to reduce the risk of negative effects resulting from unexpected behaviors of your application at the lowest possible cost. So let’s talk about the various costs associated with testing code in cloud applications.

Why Is Continuous Integration (CI) Testing Important?

Testing during the continuous integration phase of the application development life cycle is a lifesaver: It enables you to fix problems even before they occur, which prevents production issues that would affect your end-users, customers, or your business. Additionally, testing lets you speed up the continuous delivery process and reduce integration failures.

Known Costs of CI Testing

Issues in production can be extremely costly, but no one wants their end users to be affected by issues in their applications. That’s why we use software tests: to detect errors, defects, and bugs as early in the development cycle as possible to minimize the cost and create a better experience for end-users. However, tests do have some known costs, such as the following:

Building cost: This is the combined cost of time, money, and effort being spent to create tests specific to your applications. It also includes the cost of making your system available for running tests.
Running cost: The cost of the compute resources during the runtime duration of the tests.
Sustainability cost: The cost of changes needed to adopt the tests to the software alteration.

Software teams spend many man-hours and cloud resources on running their tests, meaning the cost of software tests should be viewed as just another cost item in the software development process.

It is possible to reduce execution costs in several ways. For example, you can decide to only run tests that affect the code that has changed, uncouple tests that overlap with each other, replace slow input data with mocks or test doubles to speed up testing times, or stop a test suite right after it proves there is problematic feedback in order to apply the fail-fast system.

During different runs, some tests can both fail and pass with exactly the same codebase. These are known as flaky tests, which are more costly than others because they increase the risk of a crash or slowdown of applications while also introducing ambiguity. If flaky tests are not dealt with in the organization, then developers start to feel the ramifications of ignoring them.

Since flaky tests have a high potential to badly affect your end users, you should measure and reduce them to a minimum when possible. Tests can be flaky up to a certain degree in cases such as I/O operations that are required to run. In these cases, rewriting tests to prevent flakiness can be very costly (and also complex) –yet also necessary. To prevent or minimize flakiness, you can add retries or strengthen the I/O operations against transient failures to prevent or minimize the flakiness. Additionally, you can accelerate the speed of your tests by recording and replaying the traffic using mocks for external dependencies.

Unknown Costs of CI Testing

Testing will never be able to tell us that an application is 100% reliable because testing is about managing risk. The coverage of traditional tests should not be a target that an engineering team focuses on, as it cannot indicate the quality of service but instead only calculates how extensive unit tests are.

The chart below reflects the common problems encountered during software testing. Long-running tests, flaky tests, and tests that fail can result from unknown causes and can therefore lead to unknown costs.

How to Optimize Cost

One way to optimize a system is by balancing execution cost with feedback time. For example, if we reduce the execution cost, then it will take us longer to determine whether or not our work has been successful.

If you run tests less frequently for every code commit, then the gaps in the test execution map will increase. To ensure a project's success, you need to strike a balance between the tolerance of risk and the cost of testing.

While some parts of an application may need to be tested thoroughly, others may not need to be tested as extensively. You should always think about the best way to test your scenarios, such as new features, bug fixes, or dependency failure handlers, to keep testing costs as low as possible.

For example: Would it be enough to use a unit test? Is it okay to use test doubles and mocked dependencies in tests, or should tests run against real cloud resources? Should applications be instrumented and undergo canary deployment in production in order to make tests flawless?

A developer or a development team should evaluate these questions to determine if there is a valid enhancement on a case-by-case basis. Moreover, developers should be familiar with a wide range of testing methodologies and be confident enough to choose one or more of them to ensure the cost and performance effectiveness of tests.

How Do We Plan for Tests That May or May Not Pass?

One way that we can test for unknown problems is by introducing random data to an application. Randomized testing, as it is sometimes called, introduces data that the developer has not generated intentionally and utilizes existing tests for expected behaviors. See the chart below for more information:

When testing our application, our goal is to move as much information to the “Known Knowns” quadrant as we possibly can. On the other hand, every change we make in our application invalidates the information in the "Known Knowns" quadrant.

Testing allows us to move information from the "Known Unknowns" to "Known Knowns," because we know which questions we should ask in the "Known Unknowns" quadrant. So, when we run tests with those questions in mind, we have a good chance of getting our answers.

This can be done in a quick, automated way. Some examples of questions you might ask are, "Will this function provide the expected outcome when these specific arguments are provided?", "Does my service return an HTTP 400 status when given an invalid payload?", or "Does the UI show me an error message when I enter a password that doesn’t match what is provided on sign-up?"

"Unknown Unknowns” are the issues that become apparent only when they arise, because we are not aware of these issues and potentially, we may not easily understand them.

We cannot predict which components of our application will fail without knowing what can actually fail. The best solution to deal with this problem is to have adequate instrumentation and good debugging tools in place.

It's always a good idea to go for the least costly tests to avoid regressions and to move information to the "Known Unknowns" quadrant, especially if the root cause is not just a transitory operational issue, but something that we are likely to encounter again in the future.

Conclusion

As organizations move to the cloud and adopt Agile methodologies and DevOps, developers are increasingly responsible for more tasks. No longer dedicated QA, developers write more tests, are responsible for operating services (YBIYRI), and are expected to do more for testing, security, etc.

The most efficient way to build resilient, performing, simple/easy to debug, and easy-to-maintain applications is by testing your applications. Testing enables organizations to ship code faster, but when you don’t do it carefully, it can be costly in many ways. Modern applications are distributed, dynamic, and remote. As a result, new failure modes have entered the scene and they are making work increasingly difficult to anticipate and troubleshoot.

The best way to avoid software faults is to use more than one quality control strategy. In order to accurately identify, diagnose, and correct problems more effectively, it's important to review traditional testing techniques and adopt new ones for every stage of the development life cycle. Testing will not be an effective means of low-defect software production until this is implemented.

At every testing stage of every software development project, it’s highly potential to face “Unknown Unknowns.” You need to have the proper tooling in place to immediately tame those monsters. Foresight enables you to monitor your CI pipelines, and debug and troubleshoot your tests.

Detecting the root causes of erroneous tests and understanding the reasons behind slow builds is easy with Foresight. Sign up for Foresight and try it yourself.

Difficult Mock Life

Ismail Egilmez — Mon, 12 Sep 2022 13:57:46 +0000

What Is Mocking?

Mocking is a popular and widely used approach in unit testing to handle dependencies. However, this method has both an upside and a downside, so it’s important to compare these pros and cons to determine if the costs will outweigh the benefits. Let’s take a closer look at mocking.

Mocking is a great way to isolate tests from external factors, like databases or web services. However, mocking also adds an extra layer of complexity to our codebase that must be carefully considered.

Mocking is also a useful way to replace dependencies with stand-ins in your unit test. The stand-ins are often called “mocks,” and they allow the unit test to run smoothly without invoking the real dependency.

Mocking can be achieved in various ways. The most popular approach is to create a mock object that implements the interface of the dependency, like so:

`public class LoginServiceTest {

@Test
public void happyPathLogin() {
    // Mock a User object
    IUser sampleUser = mock(IUser.class);
    when(user.passwordCheck(anyString())).thenReturn(true);

    // Mock database connection to find the mocked user
    IUserDatabase database = mock(IUserDatabase.class);
    when(database.findUserByEmail(anyString())).thenReturn(sampleUser)

    LoginService service = new LoginService();
    service.login("mail@example.com", "password");

    verify(user, times(1)).isLoggedIn(true);
}

In the example above, the class mocks a user object and a database connection to satisfy a successful login event when invoked with certain parameters.

Mocks originally generated from “test doubles,” but mocking is more widely known because it became a generic term among developers. A mock stands for the real production code in a unit test and should be able to produce assertions about the manipulations made by the test subject during the test run.

When mocking dependencies, it’s crucial to consider how much time and effort it will take to set up the mocks in tests.

The Problem with Mocking

To put it simply, you basically create a living space for the bugs in your application when you mock dependencies – meaning you override the business logic with your mocked classes where coupling with the test subject may vary.

There are three issues to consider while using mocks in your unit tests:

The complexity of your architectures can increase.
Refactoring your code becomes more difficult.
The likelihood that bugs will be hidden behind the curtain increases.

1. Reduced Simplicity

Mocking can reduce the simplicity of architecture design by coupling units together more tightly and by raising the difficulty level for making changes, which comes with a cost. When the system is harder to change, the design slowly starts to deteriorate – and refactoring becomes a hassle for developers.

Mocked resources add additional coupling between the test and the source code because mocks don’t just change the state or what is returned – they assert how an object behaves toward its collaborators. Additionally, since the internals of how the class interacts with its mocked resources are exposed, tests produce encapsulation. Thus, the test would fail if the interaction between a class and the mocked resources is changed when refactoring.

Moreover, an exception script has to be written in a mock-based unit test. To do that, you need to have a deep knowledge of the interaction between your application and its dependencies, which are generally cloud resources. It’s not enough to have knowledge of the sequence of calls – you also need to know about the data that is both inbound and outbound.

2. Difficult Refactoring

Most of the time, refactoring results in quick solutions when you look at your code from a divide-and-conquer point of view. You can see positive changes in many areas as the code quality increases.

But mocks inhibit refactoring because of the coupling they add between the test and the source code. If you are making an Interaction-Based Test, then coupling will increase between the test and the source code.

In a red-green-refactor cycle, it’s essential to see the green light before attempting refactoring. And it is what we expect to be in the “green” state after the refactoring, too. But if you mock your cloud resources for your unit tests, this expectation may fail because the mocks test both the internal and external behavior of the code.

This always happens to me whenever I decide to use mocks for small jobs. I have a UserSave class, which calls a database cloud resource and a notification service cloud resource. To make things quick and easy in my local environment, I mocked those dependencies as UserDatabase and UserNotification classes.

But then I realized that the design would be simplified if I refactored the application in the latter way, as shown in the image above. The functionality of my classes did not change, but the external behavior completely changed. So after this refactoring, many of the UserSave class unit tests have failed.

When you use mocks, your test code knows the internal requests and responses of the test subject classes. And that’s the reason why mocking prevents refactoring – because it tightly couples the production code with the test code.

3. Bugs Can Easily Hide

A good developer knows that mocking is a double-edged sword: It can be used to save time when testing the logic of your code, but there are some pitfalls that should not be ignored. The problem with mocking is that you are overriding the logic of the mocked class. The real logic gets hidden behind the scenes, which is where bugs just love to live.

Consider this: The mock may have attributes, methods, or arguments that the real object doesn’t have. Also, the return values can be different. Your unit tests run with mocked resources, and you decide the return values you will get.

On the other hand, real cloud resources may return different and/or unexpected values. The mock’s side effects and behavior may differ from those of the real objects. For example, when your real cloud resources raise an exception, the mocks might fail instead.

Photo by Thomas Bormans Unsplash

You can essentially create a mock “jungle” when mocking the interactions between the test subject application and the cloud resources. Mocking all of the classes makes you create mocks that return other mocks. If the data pathway is long and complex, then you have to mock all the way down from end to end.

This creates a perfectly ideal habitat for “cute” little bugs because:

Mocks must be updated as frequently as the application code changes, therefore adding a maintenance burden to the application. If you don’t update the mocks, then you open the door for bugs.
Integration-Based Testing makes refactoring difficult because the mocking frameworks introduce tighter couplings between test and source code. This may lead to bugs after refactoring.
Most of the errors and bugs can be hosted as a result of Integration-Based Testing. Problems arise from coupling introduced by poor implementation in unit tests, such as complex constructors, mocking value objects, etc.

When to Mock

This isn’t to say that mocking is bad or harmful. But it’s best to mock dependencies only when you really have to, not just because you can.

It is a good practice to demarcate groups of objects with mocks using state-based tests internally.

Robert Cecil Martin, colloquially called "Uncle Bob," says in his blog: “Only use a mock (or test double) when testing things that cross the dependency inversion boundaries of the system.”

First of all, if I really need a mock, I write it myself rather than using a mocking framework. I believe the simplicity of the code is only minimally affected by doing this. When I need to write test doubles for large interfaces or third-party libraries, however, then I prefer to use the mocking frameworks.

And also, when I need to mock a resource, I first try to mock at the highest level possible in the class hierarchy diagram. To be more specific, I tend not to use a mock if a spy will get the job done. Similarly, I do not use a spy if a stub will work, and so on. The idea behind this approach is that the lower you go in the class hierarchy of mocks, the more knowledge duplication you create.

We all get frustrated when our unit testing goes beyond the bearable limits in regard to time. When mocking is not used at all, the execution of the test suite can be very slow – it can even take hours. Databases, different kinds of servers, and services run thousands or millions of requests over the network, a process that is considerably slower than computer instructions.

The tests are sensitive to faults in sections that are unrelated to your test subject. Such situations can be seen in many instances. For example, databases that contain extra or missing rows. Modifications can be made to the configuration files. Network timings can flicker due to an abrupt load on the hardware. Memory might be consumed by some other processes. The test suite may require special network connections that are down. The test suite may require a special execution platform, similar to the production system. And this is where we see the benefits of mocking because it can save you from such cases.

Efficiency Vs. Cost

Overall, the effectiveness and the efficiency benefit of mocking are lighter than the maintenance and development cost. My two cents would be to consider thriftily mocking cloud resources.

If you design your application architectures and find ways to test them that do not require mocking, then you should use mocks only for edge cases to test architecturally significant boundaries.

Also, mocking tools might fail in your testing somewhere in the process, so it’s better to depend on them as little as possible. Another note: Writing your own mock classes will help you maintain as much control as possible, while third-party tools will limit your ability to keep control.

Testing > Mocking

Testing your software helps you reduce your costs in multiple aspects. There are various ways to test your software. Using mocked versions of the cloud resources may help you reduce complexity and cost if it is done in the right way.

Overall takeaways on mocking are:

You don’t have to mock everything.
Always think about the cost of development and maintenance.
You should only mock the class that is under your own control.
Only mock tests’ relevant behaviors.
Avoid mocking value objects.
Avoid mocking complex setup or constructors.
Write your own mocks.

As a final call out, remember that using mocks can bring a cost advantage for testing your applications but this is more like an illusion. Because you can not oversee the cost of potential defects in your application when you deploy it to the cloud environment.

Foresight comes into help at this point. It allows you to develop, test, and serve your application in the real cloud environment with the confidence of error detection. Foresight provides full visibility and deep insights into the health and performance of your tests and CI/CD pipelines. When mocks work in your local but cloud resources don't act the same as mocks, you can just plug Foresight and pinpoint the issues easily.

To try yourself; sign up for your free account, take a look at the documentation, and join our Discord community for any questions.

(Series #2) Building a Test Automation Team

Ismail Egilmez — Mon, 12 Sep 2022 13:56:08 +0000

We talked about topics to cover in our test automation strategy document in the first article of this blog series. In this second article of the series, we will cover some ideas about building a test automation team within a software organization.

The other articles cover topics as below:

Volume 1: Building a test automation strategy
Volume 2: Building a test automation team
Volume 3: Choosing the right testing framework
Volume 4: Knowing when to automate tests

The Need for a Test Automation Team

Test automation is a piece of code that tests other pieces of code. For that reason, automating tests in software organizations require people with technical expertise and skills in coding.

Writing good unit tests and providing good code coverage is possible with commitment and discipline, but a large investment of time and effort is required to create more complex integration tests or acceptance tests. If your developers are too busy to create quality tests, then it’s much better to build a team whose sole job is dealing with tests.

Today, we can automate many of the simpler, common tests performed for applications. And it’s a good thing – the need for test automation has increased in software organizations that embrace DevOps and agile methodologies. Automated testing is a great way to deliver more resilient code more quickly, and leave technical resources for more complex issues to be manually tested.

A test automation team works like its own company within the overall company. It owns the responsibility for designing, creating, and maintaining test automation in the entire organization, and consistently checks and validates product quality, and gives continuous feedback to development teams.

Roles and Responsibilities in the Test Automation Team

The development of robust testing infrastructure for applications is challenging due to the facet of constant change. The source code of applications is a living asset because it constantly changes, evolves, and grows as product management creates new feature requests as a response to users’ feedback.

For a company to take software development seriously, they need to actively maintain the source code of applications. Responsible development teams will view the source code as a high-value repository that must be carefully managed and maintained.

The code for any kind of tests, whether they’re automation tests or manual tests, should be taken just as seriously as the application code, and should likewise be maintained as a living asset in conjunction with the application code.

Every development team works to write quality software, and development managers apply various methods to keep discipline high enough to deliver the utmost quality in their products. It is critical to cultivate and maintain the same mindset in the test automation team.

The manager naturally should look for developers who can write automated tests as they create their automation team. But then the need for specialized professionals arises, making the automation team critical to application success. Below are some roles and responsibilities that cover the general requirements of this unique team.

Team leader

The team leader must not only develop a test plan but manage the workload of their team members. They also coordinate with other teams to complete testing tasks. The team leader must have authority over assigning and controlling workflows for members of their team.

Developer/Tester

Test developers are responsible for developing and executing test cases. They are the experts in the functionality of the application and are responsible for analyzing and reporting the automated test results. They also know how to develop tests in any format, be it data records or scripts within a framework.

Infrastructure engineer

Specialists on infrastructure generally focus on maintaining the health of the test automation framework, test management, remote execution of tests, and the development of the automation.

Automation experts

Experts must have administrative-level access in order to successfully execute automation tasks. These tasks include initial installation and configuration of the tool, as well as designing and configuring the test task for a given project. These experts are also imperative in upgrading the test automation and environment when necessary.

Team Culture

These guidelines are ideal for establishing a strong automation team. If these guidelines are followed, it is very likely for teams to achieve success in the long run.

From the veterans to the junior members of a test automation team, every member should collaborate with each other and contribute to the test automation code. By doing so, everyone takes full ownership of the code being produced.
The test automation team manager should act as a mentor to upskill every team member in their specialized areas.
The test automation engineers should be able to work with production code when it is necessary.
Developers or testers should only modify production code if they feel confident enough to automate tests.

Of note, having only one test automation engineer on the team is unsustainable. If possible, assign at least one person to be the designated maintainer of automation that has been built. This role can rotate on a daily basis.

An engineer should be present in the team to manage the automation framework, while another professional should be keeping the entire team up to date by looking at the versions of the tools being utilized and the languages being used.

Team managers should monitor automation health and its usage and should be able to point out areas where tests are running longer than expected, are flaky, or fail frequently. Tools like Foresight help testing teams resolve issues pretty swiftly.

How the Team Evolves

While considering new team members to join the team, take a close look at those who are motivated to learn and conform to the approach the team has already decided upon. Investigate and learn about their experience with other teams and where they contributed their best work on those teams.

After the interview process has been completed and you’re considering who to hire, remember to be cautious of those who say they knew better than their previous leaders about what was best for their team’s success. Ask about practices they picked up from previous team members and leaders that they have applied in the past.

It’s important to find people who are open-minded and not closed off to new ways of doing things. You should also look for employees who align their work and motivation with the team’s and organization’s direction, and who don’t take their own approach without direction.

There’s always room to evolve over time and improve, regardless of what your team set-up looks like. You should never forget that automating tests help the organization save time, money, and energy, which can then be funneled into application development.

4 Software Testing Roles

Ismail Egilmez — Mon, 12 Sep 2022 13:54:39 +0000

Software testing is the process of verifying a program or application to ensure that it performs as expected. It is an important part of the software development life cycle, as it can save teams from costly fixes. By understanding when and how to effectively run tests, a team will be able to avoid common pitfalls in the software development life cycle.

Software testing teams play a significant role in determining the success of a software product after development is completed – otherwise known as the pre-production stage.

That's why the players of the software testing team should be highly talented and capable professionals in their domains – to put the bar of success as high as possible.

As experts in this field, they should be able to create test scripts that will identify problems with the product in question.

Categorization of Roles in Software Testing

Every organization has its own team structure, but there are a few positions that need to be filled either by role or responsibility. These positions are critical to the success of testing teams because they cover different aspects of the testing process. These are:

1. QA Engineer

This position generally covers more than testing processes. A software quality assurance engineer constantly monitors each and every phase of the software development process and makes sure that the developed software meets quality standards. Additionally, they make sure that the software products work seamlessly without errors before they are pushed into production.

2. Test Manager

A test manager acts as a project manager. This is a management position within the QA or test team, which is very common for custom software outsourcing organizations.

3. Test Engineer

This is generally used as an umbrella term to cover many capabilities. It can refer to many engineers specialized in various testing approaches, such as manual testing, exploratory testing, performance testing, etc. It is also widely used to infer a testing position that minimally relies on automation.

4. Test Analyst

This is a position that, rather than being more technical, focuses on business problems. Test analysts ensure the functional readiness of the application is acceptable before it is pushed into production. They generally design, develop, run, and troubleshoot tests to catch any defects or errors in the code in pre-production environments.

5. Test Automation Engineer

This is a widely spread position among enterprises representing an engineer who codes (most likely a developer), but whose sole focus is on automating test processes. These people use testing frameworks such as Selenium, Cucumber, or others to effectively design and write new test cases. Another great advantage of test automation engineers is that they are well versed in GUI design and software testing.

Core Functions, Skills, and Responsibilities of Roles in Software Testing

The software tester’s role in a project is utilized differently in different organizations depending on the size of the testing team, structure of the team, and specific requirements of the organization.

Software testing job titles or positions are typically called “QA” or “test engineer”, but the technology they use, the business domain they are in, the expertise they have, or even the type of testing they make will differ.

In the table below, you can review a summary of the core functions, responsibilities, required overall skills, and knowledge of required tools of some common software testing roles.

Overview of Software Testing Team

The level of quality that can be achieved and how quickly you can reach your testing goals is entirely dependent on your testing team’s ability.

It is important to have the appropriate balance of different types of testers to have a testing team that effectively complements each other. Another important topic when creating a software testing team is to gather together domain experts who have all relevant required knowledge to test an application thoroughly as a team.

A strong software testing team needs to be structured in a certain way: The hierarchy needs to be clear and responsibilities need to be well-defined among team members. Well-organized teams are capable of handling all of the work effectively.

If each team member knows what is expected from them, then they will be able to complete their work as needed within the deadline. Measuring the performance of the tester is key to success. You should be able to understand what kind of defects the tester is able to uncover and which ones they may miss.

A “roles and responsibilities” document, which clearly defines everyone’s duties, should be created in participation with all the testing team members. This can be useful to aid quick communication with the person responsible for a task when an unexpected issue arises.

Do We Still Need Testing Roles, or Should Developers Do Their Own Testing?

Developer-based testing is being widely adopted by small-, medium-, and large-scale companies. Software developers are now required to write/run their own unit tests in many organizations. Moreover, some are asked to write automated and integrated code-based tests.

However, building development tests takes up time that developers could be using to produce new code. As a result, many developers skip this process and create features without testing them.

Developers can provide a significant amount of test coverage for their code, but this is often difficult to maintain if they don’t do it consistently across the entire codebase.

Some software development teams release code into production with only a couple of unit tests or code reviews. There are some teams that don't even check to see if the code still compiles after making changes. Thus, to avoid consequences, if you are going to apply developer-based testing, you should have a granular, well-defined, and well-managed process for testing.

There are software development teams that want to bypass testing their code for several reasons. One of the biggest reasons is the lack of visibility. When teams work within a small scope of the project, just like in most microservice applications, they are not able to understand the full execution flow of the entire application. This leads teams to be unwilling to test the produced code.

Some developers strongly think that creating unit tests for their code is nothing but a waste of time: It's complicated, boring, and difficult to execute repeatedly, even if test automation is in place.

You may ask why developers would test the software they create. There are developers who test their code thoroughly and take full responsibility to own the application. They write their own tests to understand if a code-fix breaks the production code, or even the build or the functionality of the application.

It's not always easy to find these people who take testing as a matter of pride and cannot go to sleep without making their code flawless in pre-production.

But here’s why testing is so important: Writing tests save time in the long run by reducing issues both in pre-production and production environments. Also, end-user/customer satisfaction increases because developers start to catch defects before they reach end-users.

Many developers don't like testing because it can be tedious, repetitive, and complicated. They do not fully understand the nuances of how every aspect of the application works together.

However, if you manage it well, in other words, if the software teams get round to testing as well as coding in their development life cycle, then the developers’ testing is very productive and indispensable.

Foresight helps developers, test engineers, QA engineers, QA managers, test managers, test analysts, test automation engineers, engineering managers, and all with its test monitoring and debugging capabilities.

If you haven't yet, get started with Foresight today!

Why a CI/CD Pipeline Makes Good Business Sense

Ismail Egilmez — Mon, 12 Sep 2022 13:51:54 +0000

If your business is thinking of switching over to an integration and continuous deployment/delivery (CI/CD) model of software production, the first thing you may wonder is whether this move is good for business. Migrating and training your team will take time and money, but sometimes it’s not clear what the payoff is and how and when you’re going to start seeing any ROI.

It’s reasonable to be cautious and not leap to embrace whatever the latest buzzword happens to be. But CI/CD has been around for the better part of 20 years and has proven its value as part of the DevOps culture, which encourages a much faster release cycle and is more responsive overall to business needs.

CI/CD has also managed to prove itself not only in terms of speed—which is not in itself a metric of success—but in terms of driving concrete business value, such as shortening time to market with new applications, features, and services, as well as improving metrics like customer satisfaction and loyalty.

That’s why it’s important for business decision-makers to push past the hype to fully understand from the ground up what CI/CD actually is and what it has to offer an organization. That way, there will be maximum buy-in—not just within IT and operations, but throughout the entire organization.

To help your organization get informed before deciding whether or not to invest in CI/CD, let’s take a look at some of the benefits and challenges CI/CD pipelines introduce.

CI/CD Pipelines: The Basics

Understanding the CI/CD pipeline requires a shared grasp of some foundational concepts and terminology:

Continuous integration. As code is changed, it is automatically and seamlessly merged with the central repository, or “mainline.” This is done frequently to eliminate the possibility that some or all of the development team is working with out-of-date copies of the code, which could cause an application to break.
Continuous delivery. When checked-in mainline code is ready for deployment to end-users, it is then alpha and beta tested in “staging,” or non-production environments, and then manually released to production.
Continuous deployment. While with continuous delivery, customers do not receive the update until it is manually released, continuous deployment extends the use of automation in the testing environment and automatically releases the newest changes to production as soon as they have passed all required tests.
DevOps. This term is ubiquitous in the CI/CD world and is often treated as if the two were identical. Dev/Ops refers to the underlying mindset more than to a particular methodology. It refers to the cultural change required to foster a faster release cycle, create collaborative interdisciplinary teams (comprising developers, operations, quality assurance, and management), and introduce automation into the process.

Before CI/CD, software development generally took place according to the “waterfall” model, where each stage took place completely separately. Code for new features, services, libraries, or applications would be developed and perhaps unit-tested separately. Then, they would be integrated, potentially creating a large number of failures all at once. Then, pre-release testing would take place simultaneously near the end of the software development life cycle (SDLC).

It’s easy to understand how, with this model, moving an application from development to production was a major undertaking that companies were not willing to do very often. This is especially true because, with each release, new bugs would be discovered—since regardless of which model you’re using for development, it’s impossible to release error-free software.

With the waterfall model, discovering a bug once the software was in production often required taking apart the software to track and troubleshoot, potentially requiring the company to begin the cycle all over again.

By focusing on smaller, incremental changes to code, and a faster release cycle, CI/CD provides greater agility. With this approach, if bugs are discovered in post-production, a mechanism is already in place to perform rapid fixes or implement new features, leading to greater customer satisfaction. Over the last decade, study after study, such as this recent research article from the University of Johannesburg, has shown vastly more positive outcomes for this style of software development over waterfall-style development.

Today, organizations using a CI/CD approach generally release daily or every few days—but new releases can occur multiple times a day if needed.

Together, CI and CD are referred to as a “pipeline” because the goal is to keep functional, fully integrated code flowing through the pipeline and ultimately into the hands of end-users.

Many large vendors are now offering tools for CI/CD software development: Microsoft, AWS, IBM, Red Hat, and others. But since organizations’ needs vary widely, most end up creating a custom “tool stack” rather than embracing an end-to-end solution from a single vendor. Tools required include source code management (such as GitHub), a CI server (such as Jenkins), provisioning (such as Ansible), and deployment (such as Terraform).

One final term you may encounter that is also important to understand:

While DevOps and CI/CD are not synonymous, setting up a CI/CD pipeline is by far the most common way businesses can achieve a DevOps transformation in modern software development.

Business Benefits of CI/CD Pipelines

When it comes to business benefits, we need to cut through the high-tech buzzwords and focus on concrete deliverables that take the company strategically in the direction it needs to go. Here are just a few of the benefits companies have found from implementing CI/CD:

Better quality software. With CI/CD, your developers are handling smaller sections of code, meaning that if problems do emerge, it will be simpler and quicker to find a resolution. It also lowers the chance that errors make it into the production environment. Additionally, by incorporating automated testing, CI/CD can resolve many bugs before they make it into end users’ hands, boosting user satisfaction.
Faster time to market. Whether your product is new or an update, CI/CD gives you a literal pipeline straight to your market while ensuring that thorough testing still takes place. And since requirements can change often during development, these changes can be incorporated easily, leading to a better-quality product once it’s released. This means you’re going to start seeing value sooner as well as maintain a competitive edge.
Fail fast. It may sound undesirable to hope something will fail quickly, but this is actually one of the most important principles of CI/CD. When new functions and features go wrong, you’ll get immediate feedback and can fix the product quickly. Because of this, CI/CD is the perfect way to conduct various types of tests: A-B testing, canary launches, blue-green testing, and dark testing are all ways to deploy and test new product features prior to a full release. And if there’s ever a problem with a release, CI/CD guarantees that you can roll things back without any serious customer impact.
Smashing silos. While the term “breaking down silos” is everywhere in today’s business world, CI/CD is probably its biggest use case. By creating collaboration among developers, product managers, testers, and operations admins, you’ll build a culture of agility and innovation throughout your entire organization along with boosting shared responsibility, keeping everybody on your team aware of the business mission and strategy.
Happy workers. Coders love to code. Everything else is secondary, and many functions development, IT, and operations teams perform are tedious and can actually lead to burnout. CI/CD introduces numerous points where automation can take over, giving them more time for creativity and enhanced productivity, which leads to cost savings.
Data stream. With its modern, business-oriented tooling, CI/CD offers the flexibility to generate, track, and report on valuable metrics across the entire SDLC. Some tools even provide ongoing monitoring and observability data from the production environment, which help your business leaders make important decisions when it comes to infrastructure, product roadmaps, team performance, and more.
Ensuring observability. This is not only a function of CI/CD. Essentially, modern software development environments are distributed, meaning they may be spread across physical locations or in the cloud. With code changing often, it is possible that something that previously functioned properly may stop working, and then it can be tough to track down the issue that caused the application to break. This challenge can be solved by building observability into the CI/CD pipeline with tools that can correlate application performance with code merges to help you speed the time to fix problems.
Too agile? Because a well-oiled CI/CD pipeline provides such superior agility, it may be tempting for execs and managers to demand one-off changes, potentially just on a whim, introducing features or “fixes” that don’t fit the organization’s overall strategy. Constantly introducing changes without keeping an eye on the strategic roadmap can make it tough to achieve the cost benefits promised by CI/CD.

Business Challenges of a CI/CD Pipeline

Let’s be honest, any type of change within an organization is difficult. While the development team will likely be satisfied overall with the benefits of CI/CD, here are a few challenges you could possibly encounter.

Beyond technological issues, CI/CD may raise cultural issues within your organization. For instance, developers and admins may fear that they are likely to put themselves out of a job thanks to all of its automation benefits.

It’s best to communicate to your employees that the move to CI/CD is part of a commitment to the DevOps mindset as a whole. Explore issues like end-user benefits and help these team members understand how essential their work is to your overall business goals.

You also need to provide training wherever necessary to help employees upgrade their skills and be able to use any new tools they are expected to work with. It is generally less expensive to train your existing workforce than find new hires with expensive cutting-edge skills. For example, project managers may need support in understanding today’s agile processes and tools in order to keep up with the velocity of CI/CD. This is also part of the up-front investment of CI/CD.

Metrics for Tracking CI/CD ROI

It may be hard to believe, but each year, around the world, failed IT projects wind up costing $260 billion (€210.84 billion) yearly in the U.S. alone. How do IT projects fail? Problems encountered could include non-completion, inability to release, failure to effectively address business needs, cost overruns, and missed timelines.

Therefore, the first metric of value in determining the ROI of CI/CD pipelines is ensuring that they have cut the risk and cost of IT failure with higher-quality code and highly functional products and systems. By automating and optimizing provisioning and testing workflows, your company should begin almost immediately to be able to reduce errors and lower costs.

Key performance indicators (KPIs) for your CI/CD pipeline’s effectiveness can be measured through tracking factors like deployment frequency. However, simply releasing fast isn’t an accurate indicator of effectiveness, so this metric should be combined with others such as deployment failure rate, change volume, the ability to meet availability and performance SLAs, mean time to recovery (MTTR), defect volume, and escape rate.

Conclusion

Earlier, we mentioned that CI/CD is not synonymous with DevOps. Yet the benefits of both are clear. While the older “waterfall” method encouraged a slow and cautious mindset—since a software release was by necessity a major and often traumatic event—today’s agile development models foster a mindset of speed and quality.

That mindset in itself can help your business tremendously. Developers who observe how simple it is to iterate the application will be quick to suggest improvements or flag problematic behavior before it lands in front of a customer.

As so many companies have discovered, a strategic move to CI/CD can offer you more innovation overall, along with clearer communication and better productivity throughout the entire organization. Which ultimately gives you a competitive edge.

Building observability from the ground up with Foresight can help overcome many of the main obstacles on the way to a productive CI/CD pipeline environment, including the challenges of observability when debugging highly distributed microservices and cloud-native applications. Find out how easy it is to get started with Foresight today.

We will be glad to answer your questions through our Discord community, support@runforesight.com, or our contact us page.

Introduction to Software Testing

Ismail Egilmez — Mon, 12 Sep 2022 13:50:00 +0000

Cloud-native applications provide benefits such as rapid deployment, scalability, and reduced costs. The downside is that testing, debugging, and troubleshooting on these types of apps is more difficult than on traditional applications.

In recent years, software testing has matured from manual testing to automated testing. There are multiple testing tools, strategies, and patterns in the market, which are very useful and important in making cloud-native application development fault-free. In this article, we explain the basics of testing cloud-native applications.

Let’s quickly bring up what is covered in this article:

Why do we test applications?
Performance
Operativeness
Resilience
What Are the Testing Concepts?
TDD (Test Driven Development)
BDD (Behavior Driven Development)
What Are the Testing Patterns?
A/B Testing
Test Doubles
What Are the Testing Types?
Unit Testing
Integration Testing
Load Testing
Regression Testing
Testing Never Ends

Why do we test applications?

From a traditional software development standpoint, testing occurs after development is completed and handed over to test engineers by software developers. Test engineers then perform their tests and, if everything works well, the code is deployed to production.

Errors can emerge at any phase of the development life cycle in a software development project (and while there are various types of software bugs, only a few of them are known to be irreparable). It is, therefore, crucial to perform software testing to prevent the occurrence of software bugs in critical environments like production.

Quality assurance plays a big role in minimizing the chances that the production code has functionality or design errors – and software quality is assured by testing in the cloud-native era, which has become an integral part of the development cycle. Although testing does have a cost, it is incredibly useful because the cost of production failures would be much higher and impactful.

Any development team that owns an application would need to ensure service quality in terms of the following aspects:

Performance
Operativeness
Resilience

These are the bare minimum requirements to ensure software quality. Depending on the requirements of your business logic or your market-specific requirements, you might want to consider testing for security, availability, usability, UX/UI, etc.

Performance

The performance of your application as it’s perceived by the end-user is affected by complex combinations of environmental effects, access patterns, and most importantly, the data being processed. That’s why it’s so unfortunate that many software teams do not actually test for performance as much as they could, since it is a vital quality aspect.

There are a lot of test concepts, patterns, and types that can improve the performance of our cloud-native applications, but it’s crucial to construct and follow a testing strategy that aligns with the development life cycle. As an example, benchmark testing is one of the most preferred and economical ways to collect performance data about our application code.

If you don’t test your application in production, then you can’t be sure that performance regressions won’t occur. Even if you heavily invest in testing in pre-production environments, you should still expand testing practices to the entire software life cycle to ensure quality with less cost and effort.

For instance, debugging and performance management tooling is an excellent testing strategy to utilize in production for detecting and understanding issues in your distributed cloud applications.

Operativeness

Running unit tests and integration tests can help you ensure that your service performs the way you want it to without defects. In these tests, we generally define a set of expected responses from our service to make sure it operates well and correctly. However, unit and integration tests aren’t the only tests available – many others, such as mutation tests, fuzz tests, etc. can be utilized to test the accuracy of application behavior at a wide range.

As mentioned before, though, you should extend your testing strategy to production to make sure the operativeness and accuracy of your service is as perfect as it can be. You should also be agile in receiving and incorporating error feedback with functional debugging and troubleshooting capabilities. But if the application is on the cloud, there’s no guarantee that it will act flawlessly in production. The end-users, therefore, will be your best testers.

Resilience

Before the cloud-native era, testing for robustness and resilience was not a priority because mostly monolithic applications did not have as many dependencies and the types of errors that could be encountered were well-known. Microservice architectures in the cloud era, however, brought the concept of dependency to applications and made them vulnerable and fragile. Due to these circumstances, if your application is not tested thoroughly for resilience, then your end-users are likely to be impacted.

With a cloud-native microservice application, you need to carefully manage the interactions between your service and dependencies. Both errors and latencies might occur and thus impact the end-user’s experience.

Having testing procedures and standards in place is imperative to make sure service is resilient even when dependencies fail. For example, automated tests that simulate the known failures in pre-production settings should be applied frequently. Additionally, for errors and failures that cannot be simulated in pre-production environments, enrolling chaos engineering strategies allows you to test for these issues in production.

What Are the Testing Concepts?

TDD: Test-Driven Development

The idea of test-driven development originated in the ’70s in opposition to Glenford Myers’ publication “Software Reliability,” which asserted the “axiom” that “a developer should never test their own code.” Test-driven development (TDD) refers to a style of programming that involves a three-action cycle: coding, testing, and design. Testing in TDD consists mainly of writing unit tests and designs which contain the activities needed to refactor code.

A TDD project normally begins by defining and understanding the requirements necessary to reach the solution to a problem. Next, we write test cases to verify that the requirements are valid. And finally, we write the code that has passed the tests. Until business functionality is ensured and satisfied, this process iterates to complete the test cases and the required code.

TDD is also called “Test-First Development,” as it forces the developer to develop and run automated tests before the application’s actual code can be developed.

The image above shows the difference between the waterfall and TDD model processes.

The benefits of TDD are:

Test coverage increases. Almost every line of code is run by test methods.
Errors are localized with unit tests, increasing the trustworthiness of the code.
Tests serve as documentation describing the way the code works. Developers can understand how the code works in a short period of time by analyzing these tests.
The need to debug code is minimized.
You have the design-first point of view in software development.
Over-engineering is more easily avoided.

BDD - Behavior-Driven Development

Behavior-driven development (BDD) is a branch of test-driven development that inherits agile development principles and user stories, which presents development as a set of scenarios in which applications behave in a certain way.

It reduces the chance of mistakes and brings more clarification to development because it allows the scenarios, actions, and conditions to be transposed into a simple, common terminology between the software development and business teams.

In other words, BDD enhances communication and collaboration between product management and software development teams as it helps to describe requirements in the form of scenarios, making it easier for all parties involved to stay on the same page.

What Are the Testing Patterns?

There are a few testing patterns for cloud applications that enable us to meet quality standards in an easier way.

A/B Testing

A/B testing, also known as split testing, was originally intended to discover user responses to two different web pages that have the same functionality. Essentially, a small group of users is selected to see version A of the web page and another small group of users is selected to see version B of the web page. If the users respond in favor of a certain version, then that version of the web page is selected to move forward with.

This approach can be applied to the introduction of new features in an application in a phased way. A new service or feature is introduced to a selected set of users and their responses are measured. The results are compared to the responses of the other selected groups who used the original version of the application.

Test Doubles

Most of the time, the performance of your application under tests depends on its interactions with other components of the application being produced by other developers. A test double replaces the actual component and mimics its behaviors. The test double, unlike real components that are dependencies or external third parties, is typically a lightweight version of the components under the control of the development team.

There are many types of test doubles, such as Dummy, Fakes, Test Stubs, and Mocks.

There are some disadvantages to using test doubles, however:

The test doubles may not be available to give data to the required tests for different cases.
The functionality of the test doubles may not be suitable for testing at the time of development.
Trying to use the actual dependencies during tests may slow down the testing.

Test Stubs

A test stub holds predefined data and answers the calls it receives during tests with that data. When we don’t want to use real dependencies in our tests (or when we simply can’t), we tend to use test stubs to eliminate the side effects of the real data.

Test stubs are useful when a dependency’s response alters the system’s behavior during tests. For example, if we have a service that takes input from an external service, that input determines our service’s response. A test stub allows us to mimic various scenarios that cause a change in the behavior of the product service.

Mock Objects

Mock objects, on the other hand, record the system’s behavior and then present the recording for us to verify. Mocks register received calls, and we can then check to see if all the expected actions were performed.

When there is no easy way to verify that our application code was executed, or when we simply cannot invoke the production code, we use mocks. The downside is that there is no return value and it’s not easy to check the system’s state.

As an example, let’s take the SMS sending service in an incident management tool like OpsGenie. You don’t want to send SMSs each time you test your code, and it’s not always possible or easy to verify that the correct SMS is sent during tests. You can only verify the output of the functionality that’s done in tests. To overcome this, you can use mocks to take the place of the SMS sending service in tests.

Fakes

Fakes are objects with limited capabilities. Essentially, they work like the real object but they have a simplified version of the production code. We generally use fakes to simplify the implementation of our tests without affecting the system’s behavior.

A perfect example of a fake is an in-memory database. It’s a lightweight database that isn’t used in production, but is still perfectly adequate to use as a database in a testing environment.

What Are the Testing Types?

Even before cloud computing became popular, software testing types were known and applied in software development. CI/CD pipelines used in DevOps or Agile methodologies made it possible to automate these testing types so that they could run each time a code build was taken or a check was required.

Unit Testing

Unit tests are used for individual units and functions of an application’s code. In cloud-native applications, the essential purpose of unit tests is to test the smallest possible part of the application. It is possible with this testing type to isolate the dependencies of tested services using mock objects or test stubs in order to focus solely on testing applications.

Unit testing aims to test each class or code component to ensure they perform as expected. If you are a Java user, JUnit is one of the popular Java frameworks.

Integration Testing

The purpose of integration testing is to check whether the components of your application, such as different services, are being performed correctly. Microservice cloud applications consist of various software modules, coded by different developers. The purpose of this level of testing is to expose defects in the interactions between these software modules when they are integrated. Because of the focus on testing the efficacy of data communication, integration testing is sometimes called “string testing” or “thread testing.”

Although each piece of code is unit tested, errors can still exist for multiple reasons. These include:

The programming logic of different developers who are all contributing to the project may be different. This nuance in their understanding may require verification that the software modules work well together.
Exception handling is not always done correctly.
Database connections of the software modules may be faulty.
The project’s requirements may change by the time development continues.

Load Testing

Load testing is the process of simulating the demand on your system within a specific period of time, and then measuring the performance effect of the load on the system (such as error rates and response time). Load tests are sometimes called performance testing, non-functional testing, or stress testing.

Load testing is a crucial part of cloud-native application development and DevOps as it helps application teams build and deploy their applications with confidence in pre-production and production environments. It’s critical for the business component of your application to be able to scale to meet demand. That’s why load testing your applications is so important: If you don’t use load testing on your applications, your end-users can be impacted at times when demand bumps up.

Regression Testing

The existing functionality of applications should not break when a new functionality is introduced. Regression testing ensures that a code change has not negatively affected the existing features of an application. Simply put, regression testing is a full or partial selection of already-executed test cases (whether functional or nonfunctional) that are then re-executed to ensure existing functionalities are working fine. If a functionality doesn’t work as it should, this is called a regression.

Whenever we do a bug fix, change a configuration, or even refactor or enhance code , we may need to run the regression tests.

Testing Never Ends

The DevOps software development cycle has evolved to include tests at all phases of development. Testing never ends, and never should, in order to keep end-users happy.

Back in the monolith age and at the beginning of the cloud age, developers used to complete development and then hand the application over to quality assurance engineers before deployment. But now, since applications are almost atomized in the cloud and applications are more complex, distributed, and asynchronous, the old routine for testing has irrevocably changed. Deployment is much more frequent than before and automated thanks to CI/CD pipelines.

On the other hand, monitoring and observability are widely used in production environments, but pre-production environments are left out in the cold by observability solutions. To ease the load on developers, observability should be applicable to software tests for cloud applications by default.

You run several unit tests every time before you push your code, and integration tests are run by CI automatically. You continuously test for errors to make sure your service is served to end-users fault-free.

Testing helps us build more reliable, resilient, well-performing, and maintainable applications – if we have the ability to troubleshoot our tests quickly via observability. Instrumentation that enables us to troubleshoot our tests in CI environments is a valuable asset to have.

As discussed, tens of thousands of tests run automatically in a CI pipeline before a change in an application goes to production but this process is not easy as to speak. Foresight is a product that addresses visibility problems in CI environments.

Foresight is a comprehensive solution that provides insights and analytics for both CICD pipelines and tests furthermore automatically assessing the level of risk of software changes and suggesting optimization and prioritization tips for automated tests in CI pipelines.

Explore how Foresight works with your projects now.