DEV Community: Zafri Zulkipli

Parallel Behat

Zafri Zulkipli — Mon, 17 Jan 2022 03:41:08 +0000

TLDR; Our company dockerized pretty much everything, as such to tackle long waiting time of the CI pipeline, we split the big list of behat files into chunks and process it using docker in parallel using python multiprocessing.

Background:

In my workplace, we deploy on a daily basis. Each time we want to deploy, we have to wait for at least 20 mins for the pipeline to finish. On average it takes around 25~30 mins from the time we push the code to our version control, wait for it to finish running all the checks, then deploy to production.

The main reason for this delay is we have a lot of behat .feature files. We emphasize a lot on integration testing, as such the test files keep growing.

All these waiting is frustrating whenever we want something in production very fast. It's even more frustrating when we're working with 3rd parties and they have to wait for us to apply changes.

The search for cutting down time

We've search for various ways to reduce the time, some of the suggestion was:

Remove deprecated features and its .feature file
Refactor the .feature file to remove redundancy
Search for 3rd party packages that deal with parallelism

Point 1: we've been regularly doing it, but the time it reduce was not as much as we expected.

Point 2: no one really wants to take the time to find redundant feature files and refactor it.

Point 3: 3rd party packages are unable to to satisfy all our requirements.

After all things considered, we've decided to create our own parallel behat runner.

The journey to achieve parallelism

Step 1: How do we even run process in parallel?

TLDR; I don't even know myself the intricate logic behind parallel processing. What I do know is that I've been using python multiprocessing package and it worked wonders!

Step 2: Add a script that takes specific input to execute

Python multiprocessing needs a process to run. So I've created a simple bash script that takes in .feature folder path and execute all the files within it. It would basically be self-contained docker process that executes the test and die.

Step 3: Chunk it!

This part is easy, let say we have .feature folders A,B,C,D,E,F. By putting it into chunks we would have [[A,B,C], [D,E,F]]

Step 4: Putting it all together

Finally, we use python multiprocessing to run the script which takes in the chunk. By splitting into 2 sub process, we managed to get the time down from 20 mins to 14 mins. We're still in testing phase on increasing the no. of sub processes. The hypothesis is, more sub process, less execution time provided we have enough CPU core.

The code

For company privacy reason, I will not be sharing the exact code. This is the simplified obfuscated version, much of the things may not make sense to you or edge cases not handled but it's already handled in our actual version.

sub_process_test.py

def main():
    processes = []
    sub_process_count = 2
    chunks = get_chunks(sub_process_count)

    for index, chunk in enumerate(chunks):
        current_proccess = multiprocessing.Process(
            target=run_behat_for_certain_folders,
            args=[index+1, ','.join(chunk)]
        )
        current_proccess.start()
        processes.append(current_proccess)

    [x.join() for x in processes]

def run_behat_for_certain_folders(pid, folder_names):
    if folder_names:
        subprocess.call(f"./sub_process_runner.sh {folder_names} {pid}", shell=True)

sub_process_runner.sh

#!/bin/bash

run_sub_process() {
  docker-compose -f $docker_compose_path -p "$prefix" exec -T test ./vendor/bin/behat \
    --stop-on-failure \
    ./your-behat-folder/$1)

  # status check here, obfuscated
}

# docker initialization here, obfuscated

for i in ${1//,/ }
do
    run_sub_process $i $2
done

sub_process_test.py will call sub_process_runner.sh A,B,C 1where the 1st arg is the list of folder it needs to run and the 2nd arg is the sub process id.

Conclusion

We achieve our goal of reducing the CI pipeline waiting time. Hopefully this post bring some insights to the reader, whatever the insight may be.

Automated API Testing with Jenkins and Postman/Newman

Zafri Zulkipli — Sun, 19 May 2019 07:43:56 +0000

Let me start off by saying that this is purely experimental, I am not using this strategy in my workplace nor any production environment. This is me, spending my lazy Sunday afternoon playing around with API testing automation.

I got this idea to play around with Jenkins/Postman after my workplace is looking for an API testing automation tool that would run periodically and notify us immediately when something fails.

I knew Postman is perfect for running multiple API tests with it's Runner and pair it with Newman I can execute collections in headless state.

I stumbled upon Jenkins while googling away how to automate the process of running Postman collections. So now I have the means to test and automate. Below is my step-by-step setup for running Postman + Newman + Jenkins.

My folder structure

automated-test
├── docker-compose.yml
├── Dockerfile
├── postman_collections
├── .. # some other Jenkins related stuff that is not in the scope of this post

I begin by extending the official Jenkins docker image to include Newman.

Dockerfile

FROM jenkins/jenkins:lts

USER root

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
    # install nodejs
    && curl -sL https://deb.nodesource.com/setup_10.x | bash - \
    && apt-get install -y apt-utils \
    && apt-get install -y nodejs \
    && apt-get install -y build-essential \
    && apt-get install -y inotify-tools \
    # install newman
    && npm install -g newman

USER jenkins

Then build using docker-compose (since I'm lazy to type a long list of docker command just to start Jenkin server)

docker-compose.yml

version: "3"

services:
  app:
    build: .
    volumes:
      - .:/var/jenkins_home
    ports:
      - "8080:8080"
      - "50000:50000"

After that, a simple docker-compose up will suffice.

So I got my Jenkins server up and running, now I need to export my postman collections. Below is a simple API test that I made. It calls an endpoint and assert the HTTP response status is 200, pretty simple right?

I exported the collection to ~/automated-test/postman_collections/collection.json

Now, I need to create a new project in Jenkins dashboard.

The configs worth noting is

Interval to run the test (I set it to run every 5min)
Build Triggers > Build periodically > */5 * * * *
Actual test execution for the postman collections
Build > Add build step > Execute shell > newman run /var/jenkins_home/postman_collections/collection.json

Once saved, Jenkins will execute your postman collections every 5min.

And done! Of course using Jenkins just to test for Postman API collections is a bit overkill, but it does the job. If you have any other automation tool that you find suitable, do let me know since my workplace is still looking for a suitable API testing automation.

One of those wtf moments when using docker

Zafri Zulkipli — Sat, 11 May 2019 10:50:26 +0000

I've been using docker for a year and a half now. Since then I've learned many neat and cool tricks about docker. I'm gonna share with you one particular trick that I find very interesting when using docker. Take a look at below script.

docker run --rm -it -v $(PWD):/app -w /app busybox rm -rf deps

At first glance you can tell that I'm using busybox image to do nothing than just to remove my deps folder. But why should we concern about this? Well, it turns out that docker leverages root access, meaning we're basically running sudo rm -rf deps without asking for our sudo password! That's dangerous!!!

Although it is dangerous, it is quite useful as well. We just have to be careful on the way we use it that's all. Tbh, I've used this trick quite a lot in my development. The example above is actually a part of my Makefile setup as depict below

setup:
        docker run --rm -it -v $(PWD):/app -w /app busybox rm -rf deps
        docker run --rm -it -v $(PWD):/app -w /app elixir:1.6 mix local.hex --force && mix deps.get
        cd assets && $(MAKE) setup
        docker-compose build

As you can see, I'm developing an elixir application, and make setup is something you want to run quite a few times (if not just one time). So this ensure that if somebody were to clone my project, running make setup for them would be a breeze and won't have any issues regarding permission.

Anyways, what do you think of this trick? Is it good? Bad? Share your thoughts with me and if possible how can I improve my setup.