DEV Community: Burak Kantarcı

How engineering leaders can have more visibility on their GitHub Actions

Burak Kantarcı — Wed, 23 Nov 2022 07:03:17 +0000

Engineering leaders are responsible for development processes and their team’s experience. Today’s leaders in engineering must take decisions based on data. What measurements ought to be monitored as the key performance indicators for the team on their CI/CD process? Which specific facts should technical leaders focus on?

‍Serkan Ozal, CTO, and Co-Founder at Thundra says

Without a proper CI pipeline, it is impossible to send your code bug-free to production. If the failed tests running on your CI are not investigated properly, it is expected that the code will break something in production.

Common things when developers see a failed test or workflow are:

Re-running the whole workflow from the start
Try to reproduce the bug on local
Dig into the log piles

Although the traditional ways are still helping the developers, it loses so much time than expected.

‍Lemi Orhan, CTO, and Co-Founder at Craftgate mention how crucial is to be aware of the CI performance at a high development pace.

We have more than 160 workflow runs each day. It is very hard to identify when and why our workflows have failed. It is likely that one of our workflows start to run 5 times slower than usual. We needed to be alerted in some way.

Metrics about your CI and test performance attempt to guide you to a defect-free and quick software delivery when you take a broad view of the process of shipping software from development to production.

‍Stian Thorgensen, Keycloak Project Lead shares some insight about why they chose GitHub Actions over many alternatives

As an open-source project, we were using Travis CI, but we switched to GitHub Actions very quickly. It allows you to have twice as many parallel runs, for free. That’s why we have switched to it.

However, GitHub Actions fall short of answering crucial questions like;

🆘 Which tests are blocking the PR?

🤯 Which changed lines haven’t been tested?

💰 What is the most costly workflow?

🔻 Is my workflow success rate decreasing?

Solution: Foresight

Foresight brings visibility into CI workflows and test runs, enabling developers to debug CI, build, and test failures and latencies quickly. Furthermore, Foresight’s change impact analysis helps you analyze the impact of the code changes to detect defects early in the release cycle.

With the help of the Highlights dashboard, engineering managers, and CTOs are eligible to answer the below questions;

Which workflows fail frequently and need to be optimized?
What is the duration/success/cost trend in my workflows?
Where do we need to start taking action about health and optimization?

Stian tells his experience about Foresight until now:

We’ve been avoiding non-releasable junks due to the broken, unstable, and flaky tests. Foresight gives you the ability to look at more than one workflow at a time. You can easily see the tests and the results without losing your way in the log piles.

You may check the new code’s level of testing and how it affects your test results using Foresight’s PR view. The changes that prolong the merging process are easy to spot using Foresight. As a result, Foresight assists you in maintaining your contributions to your open-source project in a secure and effective way.

Lemi adds to his initial experiences:

We have many repositories including several mono repositories. Foresight helped us to monitor all the workflows, runs, and test results from one single place, to improve the pipeline by the alerts, and to save a lot of time. I highly recommend it to the community.

If you would like to watch the Foresight demo and hear more from Serkan, Lemi and Stian, don’t forget to check out the webinar recording

Create your free Foresight account and start monitoring your GitHub CI pipelines and tests. You can reach out to the Foresight team via its Discord community.

Microservices Testing: A Quick Start Guide

Burak Kantarcı — Tue, 08 Nov 2022 11:13:00 +0000

What is Microservices Testing?

A microservices architecture creates an application as a collection of services. Each microservice works independently and uses application programming interfaces (APIs) to communicate with other services. Each microservice has its own data store and is deployed independently.

Testing a microservices application requires a strategy that accounts not only for the isolated nature of microservices but also for service dependencies. This is why microservices testing processes often put each microservice in isolation to verify it is working properly and then test microservices together.

Microservices testing is becoming a central part of continuous integration / continuous delivery (CI/CD) pipelines managed by modern DevOps teams.

In this article, you will learn:

Microservices Testing Types

Unit Testing
Contract Testing
Integration Testing
End-To-End Testing

Microservices Testing Strategy

Pick the High-Risk Services
Contract Testing
Use Service Virtualization
Test to a Service Level Agreement

Microservices Testing with Foresight

Microservices Testing Types

When developing a microservices architecture, there are several tests you can run to validate your application, including unit testing, contract testing, integration testing, end-to-end testing, and UI/functional testing.

Unit Testing

This type of testing helps you validate the performance of each component (or unit) of your software. A unit can be an object, a module, a procedure, function, or method. You can define the size of the unit but be advised that smaller units are the easiest to test. Furthermore, if you are struggling to define a unit test, consider it an indication your module should be broken down into smaller pieces.

Ideally, developers should perform unit testing during the development phase, to ensure all code components are working properly and as designed. However, during rushed development or when teams are working in siloed phases, the quality assurance (QA) team may be required to perform unit testing — after the developers completed their work on the project.

Contract Testing

To ensure that services can communicate, a contract defines the results or outputs expected for specific inputs. Consumer-contract testing ensures that each service call fulfills that contract, and that, even if the service itself changes internally, consumers will continue to receive the same results from that service.

The service itself thus remains a black box. To test the contract, services are called independently and responses are verified. If there are service dependencies, they must operate as stubs that enable a service to function, but without interacting with other services. This will also prevent erratic behavior caused by external calls, putting the focus on testing a single service.

Integration Testing

An integration test collects microservices and tests them together as a subsystem. The goal is to verify that the microservices collaborate to achieve a joint operation. During the test, communication paths are exercised, making it possible to locate incorrect assumptions in each microservice, regarding interactions with its peers.

Integration testing can also help you verify the interactions between microservices and external components. For example, you can use this type of test to verify that external data stores and caches are properly integrated, respond, and perform as expected.

End-To-End Testing

The main goal of an end-to-end test is to verify that the entire application works to achieve its business objectives, regardless of the components making up the entire architecture. During end-to-end testing, the application is treated as a black box while tests exercise the system as it is fully deployed.

End-to-end testing is especially important for microservices architecture, which is composed of many parts that attempt to achieve similar behaviors. An end-to-end test can help you ensure there are no gaps between microservices and all work towards achieving their goal. Ideally, the tests should verify messages pass correctly between services, and that network infrastructure, like firewalls and proxies, is properly configured.

3 Microservices Testing Strategies

Here are three strategies you can use to make microservices testing more effective.

Pick the High-Risk Services

Continuous integration and continuous delivery (CI/CD) processes require automated testing. For microservices that must operate at a large scale, load testing is especially important, as are other types of testing that can be complex and time-consuming, such as integration testing and microservices security testing. However, these types of testing can take time and slow down the CI/CD pipeline.

The solution is to identify high-risk microservices and test them extensively at the beginning of the cycle to reduce risks later on. This may require additional time, but it will be worthwhile by reducing issues later in the process. For low-risk microservices, heavier tests can be delayed until later in the process.

Use Service Virtualization

Dependencies can create bottlenecks if the code is ready but held back because a dependency is pending delivery. You can use service virtualization to avoid this delay.

Service virtualization lets you mock out a service. First, you need to create a microservice containing the required URLs. Next, the URLs can accept data and respond according to the microservice specification. Note that response can be limited in scope.

Here is an example of an endpoint mockup response for /customers/{id}:

{  
"id": 567,  
"first_name": "Jane",  
"last_name": "Smith"  
}

While the behavior is limited, the mockup does provide the consumable functionality of a dependency still under development.

Test to a Service Level Agreement

To ensure efficiency and collaboration throughout the development process, testers, developers, and relevant stakeholders need to be on the same page. You can make sure all collaborators are on the same page by creating a Service Level Agreement (SLA). Ideally, testers and developers should work together to create this document, and implement it with flexibility in mind.

Microservices Test Visibility with Foresight

Testing microservices gets only harder as modern applications become much more distributed and asynchronous. Foresight is designed to help developers to monitor and debug your tests with plug&play ease of use.

Using these tools, developers can shed a light on the test failures running on build machines and/or during the CI phase. Sign up with Foresight to get started with a better understanding of microservices testing.

pypa has started to monitor workflows and tests

Burak Kantarcı — Wed, 26 Oct 2022 06:26:25 +0000

World famous packaging and virtual env management tool pypa has started to monitor their GitHub Actions and see the results:
https://pypa.app.runforesight.com/repositories/github/pypa/pipenv/workflow-run/180087c7-e109-37be-8402-236691c00948/9092458891/tests

They have almost 20 jobs on their main workflow. It looks very complex and windows tests looks a bit problematic 🤔

Built an observability power-up for GitHub Actions

Burak Kantarcı — Wed, 28 Sep 2022 08:47:52 +0000

GitHub Actions is great, but they fall short of answering crucial questions like;

🆘 Which tests are blocking the CI?
🤯 Which changed lines haven't been tested?
💰 What is the most costly workflow?
🔻 Is my workflow success rate decreasing

That's why we have built Foresight

Foresight 3.0 is launched on ProductHunt today. The best CI observability tool in the market!

Foresight’s aim is to bring visibility into CI workflows and test runs, enabling developers to quickly debug CI, build, and test failures and latencies. Foresight’s test gap analysis helps you analyze the impact of the code changes to detect defects early in the release cycle. It prioritizes tests according to their importance, flakiness, and failure rates allowing you to create a unique and reliable test order.

With the learnings we gained, Foresight started to expand its potential. To become a pioneer product to deliver high-quality software at speed, we are thrilled to introduce our brand new product to you.

Highlights

As code flows through your CI workflows, it is processed by plenty of different steps. Each step depends on various configurations and parts that change frequently. Even though your pipeline has a solid setup, it may slow you down while identifying the problems while you deploy to production.

A common challenge for software organizations is ensuring that all stakeholders — developers, DevOps engineers, and SREs — have across-the-board visibility into the software delivery process. The reason you need workflow analytics is to identify issues and enhance the organization-wide visibility of your workflows.

Foresight allows you to identify those in the quickest way to enable you to have better workflow operations.

Easy UI for workflow and test Troubleshooting

GitHub Actions only provide you with the logs of the workflow steps. With Foresight, you can get more. After integrating Foresight’s GitHub Actions, you get process traces to understand what is happening when you trigger your workflow. If there is a K8S configuration change and your workflow started to fail, you will gain visibility into the kernel-level information. Then, you have critical metrics such as CPU, Memory, Disk, and Network.

The more complex workflows you have, the harder it is to use GitHub Actions. Imagine that you have 10 different workflows and each of them has 10 different jobs running your tests in different environments. This complexity requires a lot of troubleshooting failed or flaky tests. Besides, GitHub doesn’t provide any in-depth information such as the performance history of your tests. With Foresight, you can easily see every information you need for your tests.

Change Impact Analysis

Foresight correlates the changes to the codebase with the test coverage reports to determine how much of the changes are covered by the tests, empowering teams with clear quality risk metrics to facilitate the right change management process. Pull requests can be automatically approved or passed through reviews depending on the quality risk to ensure only high-quality builds are moved to production.

Rules

It is currently hurdling to see the critical metrics of CI workflows and even not possible to set up rules and alerts for them on GitHub Actions.

Foresight makes it possible to automatically notify your team on critical workflow and test metrics. And make sure your CI pipeline is always green. For example, when your workflow success drops down under 80%, you will be able to get a notification automatically.

We have prepared some of the most common rule templates for you and keep on expanding the feature depth in this vertical.

Integrations

With the Slack and GitHub Actions PR comments, never miss a critical analysis. You will have Foresight's summary report for tests, changes, and steps.

Pricing

Foresight is 100% FREE for teams under 20 committers and also open-source projects. This means Foresight is accessible for most small and medium-scale organizations.

Above 20 committers, each committer will be charged $9 per month.

Foresight has flexible pricing that scales with your team. We have a committer-based pricing model. A committer is a real developer who commits a code in the last 30 days into a repository watched by Foresight. We use GitHub's official API to fetch the committers. We omit bots from your committer list.

Celebrate Foresight 3.0 with us this September

The very best CI observability tool in the market!

Check out now!

Next.js + Playwright + GitHub Actions Boilerplate

Burak Kantarcı — Thu, 04 Aug 2022 07:34:00 +0000

A comprehensive Todo app with APIs and E2E tests with GitHub Actions enabled.

See the repository: https://github.com/burakkantarci/playwright-todo-app

This project is a boilerplate to demonstrate the usage of modern stacks together. The project stack:

Next.js Todo App
Playwright
GitHub Actions
Foresight

Motivation

The tech stack market is expanding more and more and seeing everything in action is the easiest way to see the benefits. I decided to create a combination of Next.js, Playwright, GitHub Actions, and Foresight to create an E2E stack for a web application.

All the external tooling is free; therefore, you don’t need to pay anything to try this stack.

What will be the result

A Serverless Next.js based TODO application with Upstash, the simplest way of production ready web application.
Playwright testing framework so works in any browser and any platform. It is lightweight and easy to adopt. We will implement JUnit so that we can generate a test report.
GitHub Actions for creating a CI pipeline. Don’t need to explain GitHub, but GitHub Actions has 2000mins of free CI execution; therefore, it makes it easy to automate all your software workflows.
Foresight provides full visibility and deep insights into the health and performance of your tests and CI/CD pipelines. We won’t just build the stack; we are going to make sure it works fast and efficiently.

Setting up the Todo App

I followed the Upstash's guide as a starting point because it is very minimalistic and easy to understand.

Setting up a Next.js app is mandatory. If you have Next.js configured in the app, you can also use them. I recommend using the example above if you don’t have any prior setup.

Setting up Playwright

To initialize the Playwright, run the following command and select the configurations to get started:
npm init playwright@latest

Choose JavaScript (you can also choose Typescript)
Name of your Tests folder
Add a GitHub Actions workflow to easily run tests on CI.

After the installation has finished, you will have the base Playwright setup. Your test will be under the tests folder, and you can see the simple GitHub Action playwright.yml under the .github folder.

One thing to mention about the Playwright is they provide example test specs so that you can have an understanding of how it is working.

Writing our E2E test

We will add an E2E test which will add a Todo item and Complete it. This spec will help us ensure our APIs and user interactions are working.

Create add.spec.js file under the tests folder and paste the following code:


// @ts-check
const { test, expect } = require('@playwright/test');

const TODO_ITEMS = [
 'buy some cheese',
 'feed the cat',
 'book a doctors appointment'
];

test.beforeEach(async ({ page }) => {
 await page.goto('https://localhost:3000/');
});

test('add a todo item', async ({ page }) => {

 var todoName = TODO_ITEMS[0];

 // Text input
 await page.locator('#todo').fill(todoName);
 await page.locator('#todo').press('Enter');

 // Make sure the list only has one todo item.
 await expect(page.locator('.Home_card__2SdtB')).toHaveText([
   todoName
 ]);

});

test('complete a todo item', async ({ page }) => {

 var todoName = TODO_ITEMS[0];

 // Text input
 await page.click(`.Home_card__2SdtB:has-text("buy some cheese")`);
  // Make sure the list only has one todo item.
 await expect(page.locator('.Home_card__2SdtB')).not.toHaveText([todoName]);
});

Go to playwright.config.js and disable the parallelization. This will help our tests work one by one.

/* Run tests in files in parallel */
 fullyParallel: false,

Also, you can test your web application in different browsers at the same time with Playwright. For our configuration, I used only chromium, but you can enable or disable it on the Playwright config file.

Go to package.json and add a test script

"test": "playwright test"

With that command, you will be able to run the test by the npm run test command in your terminal and in the GitHub Actions.

Run your test and check whether your configuration works correctly so far. You should see an output similar to this:

Creating a JUnit Report

JUnit reporter produces a JUnit-style XML report. We will use test reporting to monitor the health of our test. This is essential for ensuring our Todo web app achieves an acceptable quality level.

Go to playwright.config.js and add the following:

reporter: [ ['junit', { outputFile: 'results.xml' }] ],

When you run your tests, it will generate a file named results.xml and it will look like this:

When there is an error, the reporter will include the error logs and messages as well.

Configuring our GitHub Actions

Push your code to a GitHub repository. The initial playwright.yml action will help us to run our test in every commit and pull request. Your configuration should look like this:

name: Playwright Tests
on:
 push:
   branches: [ main, master ]
 pull_request:
   branches: [ main, master ]
jobs:
 test:
   timeout-minutes: 60
   runs-on: ubuntu-latest
   steps:
   - uses: actions/checkout@v2
   - uses: actions/setup-node@v2
     with:
       node-version: '14.x'
   - name: Install dependencies
     run: npm ci
   - name: Install Playwright Browsers
     run: npx playwright install --with-deps
   - name: Run Playwright tests
     run: npx playwright test
   - uses: actions/upload-artifact@v2
     if: always()
     with:
       name: playwright-report
       path: playwright-report/
       retention-days: 30

You will be able to see your workflow runs as below if your Action works.

GitHub Action works flawlessly to automate the works you have done manually. You don’t need to run npm run test by yourself; GitHub Actions does it automatically when you commit a new code. However, GitHub Actions is not offering enough information about your tests and their performance. When they fail, you need to understand by finding them in a log pile in the workflow run details.

We will use Foresight to monitor our Tests. Disclaimer, I’m working for Foresight. It is free for all open-source projects. It requires a very simple configuration to start.

Configuring Foresight

You can use this guide for setting up your Foresight configuration. What you need to do is set up an account, install Foresight’s GitHub app and watch the repository that you’ve initiated for this tutorial.

After watching this repository, you need to update your YAML file. You can remove the last step and add Foresight’s test kit.

name: Playwright Tests
on:
 push:
   branches: [ main, master ]
 pull_request:
   branches: [ main, master ]
jobs:
 test:
   timeout-minutes: 60
   runs-on: ubuntu-latest
   steps:
   - uses: actions/checkout@v2
   - uses: actions/setup-node@v2
     with:
       node-version: '14.x'
   - name: Install dependencies
     run: npm ci
   - name: Install Playwright Browsers
     run: npx playwright install --with-deps
   - name: Run Playwright tests
     run: npx playwright test
   - name: Foresight test kit
     if: success() || failure()
     uses: runforesight/foresight-test-kit-action@v1
        with:
       api_key: ${{ secrets.FRS_PROD_API_KEY }}
       test_format: JUNIT
       test_framework: JEST
       test_path: ./results.xml

As you can see, we entered the format, framework, and path fields by the configuration we’ve created above.

This action will automatically send your test report to Foresight, and Foresight will analyze your tests in the most user-friendly way. After updating your YAML, your workflow will run automatically, and you will be able to see your workflow run results as below in Foresight:

It looks very similar to GitHub Actions; however, it is powered up with unique features such as Change Impact Analysis and Highlights. Unlike GitHub’s messy log view, you can easily access your test results with a clean design. You can learn more about Foresight here.

Summary

I hope this project will be a toolkit for you to start an E2E-covered Next.js project. It is a basic application with APIs, Playwright tests, and GitHub Actions.

With the help of Foresight and Upstash, you can power up your processes and get benefited from the latest tech stacks.

Does monitoring the status of all workflows / pipelines at once matter?

Burak Kantarcı — Wed, 29 Sep 2021 07:46:30 +0000

Hey everyone,

GitHub doesn't provide a dashboard for your all Actions status. I saw some developers use README with badges, but it's not helpful in a global view of a true pipeline that I think most people are used to seeing.

How do you solve this issue in your organization?

What are the issues on maintaining open-source projects?

Burak Kantarcı — Tue, 31 Aug 2021 13:26:44 +0000

Hi Dev.to Fam,

I was wondering what are the main issues you are having while you are maintaining your open-source project?

Is your contributors checking the software-quality, test-quality? How do you keep your pipeline quality?

Recently, I was digging into some public projects and seeing some badges on top of the readme files. It may turn into a garbage at some point. Do you think those badges are important for you while contributing to a project?

Really curious about your thoughts

How many test runs has your company in a month?

Burak Kantarcı — Fri, 30 Jul 2021 08:16:09 +0000

Hey QA / Testing people,

I'm wondering how many test runs has your company in a month?

Thanks!

What is the things you do when tests fail on CI?

Burak Kantarcı — Wed, 16 Jun 2021 06:29:22 +0000

Hey everyone,

Considering the importance of CI in the development cycle, gaining observability into the CI pipeline is getting more critical. Failed tests are a huge issue for the pipeline.

What is the first thing you do when a test fails on CI? Checking logs? Re-running? Or etc?

Cheers

What are your testing struggles on microservices?

Burak Kantarcı — Fri, 07 May 2021 12:13:12 +0000

What are the struggles are you having while you are testing microservices?