How to Automatically Generate Test File with AI

atikahe — Wed, 18 Jan 2023 16:17:36 +0000

Talks around unit tests and TDD in general have been quite polarizing. On one hand, it makes sense to have a measurable gatekeeper in place especially for large projects. On the other hand, it's just an awfully lot of work for such an unproductive task. Not to mention the learning curve that can vary for beginners, and often lack of documentation or learning sources for some language. But no matter one's strong opinion on the matter, at the end of the day, it's either gonna be a job done or a tech debt looming over us all.

Thankfully, the latest development on AI have made it possible to automate some part of the test writing process. Keep in mind that a lot of these tools are mostly research preview or usable prototype, which would require signing up for access.

Here's some tool you can use to write test from scratch, or refine existing test to be better:

1. Github's Testpilot

After launching their AI pair programmer, Copilot, last year, Github cited Testpilot as one of their many next exciting product. Similar to Copilot, Testpilot come as part of a VSCode extension and is still a usable prototype.

To use it, you need to signup to Github Copilot access, install Github Copilot Extension and then install Github Copilot Labs Extension on VSCode. Keep in mind that Github Copilot access is a $10 plan per month by the time of writing this article.

2. OpenAI Codex

A free alternative to Testpilot would be OpenAI Codex that comes in limited beta. It doesn't come in the form of VSCode extension, but instead a website where you can copy your code and insert the prompt to.

To access this page, you need to sign up for an OpenAI account, and then go to its playground as seen in the image above. The page provides you with many tuning options you can tinker with, but which could be too verbose for beginner. This is suitable for when you want to experiment and need more freedom with prompts.

3. Auto-test CLI

The next one would be a shameless plug from myself. Auto-test is a CLI tool that uses OpenAI Codex under the hood, but can hopefully provide a better experience as you can do the test generation process from the CLI. You only need to tell the filename, and it will generate the test file complete with all the test cases inside.

To use it, you need to install auto-test and include it in the $PATH of your machine. Then, get your API Key from OpenAI's account page and export it to your terminal profile. And that's it! You can use auto-test command and point it to any file you want to be tested.

The result of the test may vary. If the result is unsatisfactory to you, you can add a custom prompt that helps the AI better understand your code and generate better test. What has worked for me is by explaining a little bit about my code and what I want it to accomplish, usually with an example of input and output. Or just providing a summary of the code would also suffice. You can add custom prompt using the --prompt or -p flag.

If you want more freedom of the prompt and completely override the default one, you can always add --override or -o flag at the end of the command.

Keep in mind that in v0.1 version, the generated test file might lack important things such as importing dependencies or mocking external functions. You would still need to add this yourself, but, on the bright side, at least we're no longer writing test from scratch :)

A Novice's Guide to Integrating Code Coverage Report with SonarQube and Gitlab Runner

atikahe — Sat, 14 Jan 2023 12:20:21 +0000

Integrating SonarQube is crucial for identifying breaking changes and technical debt in large codebases. Automating unit tests and setting up a SonarQube job for code analysis can help with this. However, configuring the pipeline can be difficult, and available documentation may not be intuitive for those new to DevOps like myself. This guide aims to assist developers in setting up and testing CI/CD jobs on their own machine. One case example is to integrate code coverage report to SonarQube analysis in a gitlab job that could be tested locally.

1. Installing sonarqube

Visit here for a straight-forward SonarQube installation guide. Enterprise usually have a centralized SonarQube dashboard which you can connect to. However, for learning purposes we will install and run it on our machine instead.

2. Setup sonar config on the project's root

Go to the root of your project and create a new sonar-project.properties file.

# Project specification
sonar.projectKey=project-key-example
sonar.projectName=project-name-example
sonar.projectVersion=project-version-example

# Analysis specification
sonar.language=go
sonar.exclusions=**/mocks/**,**/vendor/**
sonar.qualitygate.wait=true

# Testing specification
sonar.go.coverage.reportPaths=./coverage.out
sonar.go.coverage.minimumCoverage=80
sonar.test.exclusions=**/*_test.go,**/mocks/**,**/vendor/**

Project specification will be necessary to identify your project among others that will share your SonarQube dashboard. So, make sure that the key is unique and the name is descriptive.

Analysis specification is important to tell sonar what framework it needs to use in running the analysis. The things they will analyze by default are: code smells, code duplication, and bugs. By default, coverage analysis won't be done unless we feed the report into it, which we'll do by the end of this article.

Specify the language you're using in sonar-language, in this case we'll use Go. Exclude the folders that won't be needing analysis, such as mock files or vendor files, and specify it under sonar.exclusions. You can also use sonar.inclusions to specify the folders that needs to be analyzed. But since by default we want everything we write to be put under scrutiny, only specifying exclusions would suffice. sonar.qualitygate.wait=true will tell the SonarQube analysis to wait for the completion of the quality gate evaluation before completing the analysis.

Quality gate could be another topic for another time, but to put simply, it is a set of threshold that would qualify your code to be production ready. This threshold could consist of percentage of code coverage, number of code smells, etc. Waiting for this evaluation may not be necessary at all, unless you want to make sure every deploy is production-ready.

Next is specifying the tests. sonar.go.coverage.reportPaths=./coverage.out tells sonar where it needs to read coverage report, with sonar.go.coverage.minimumCoverage=80 setting the minimum percent of coverage. You'd also want to list untested files, including test files themselves, under sonar.test.exclusions.

That's it! We're done with configuring sonar file.

3. Setup Gitlab CI jobs

Find these stages under the existing .gitlab-ci.yml in your project, or create a new one if it doesn't exist.

stages:
   - build
   - test
   - deploy

Usually, there are three main stages to CI: build, test, and deploy. But now, we not only want to do test as part of the stage, but also analyzing the codebase as a whole. You can add a new analyze stage after test, or you can put them both under one stage. In my case, I preferred grouping them together under analyze stage.

stages:
   - build
   - analyze
   - deploy

Next, we're going to create two jobs: unit-test and sonar-analysis.

unit-test:
  stage: analyze
  image: golang:1.18
  script:
    - go test -v -coverprofile=coverage.out -covermode=set ./...

sonar-analysis:
   stage: analyze
   image: sonarsource/sonar-scanner-cli:latest
   script:
     - sonar-scanner

See that both of these jobs belong under the same analyze stage, while also running on different docker images which would allow each jobs to have their own dependencies. After setting it up this way, both jobs will be running simultaneously, which won't allow us to integrate coverage report with sonar analysis. Instead, what we want is to run unit-test first, generate coverage report, and then run overall code analysis on top of the report.

To do this, we're going to add the artifacts.paths to save our generated coverage report. Then, tell sonar-analysis job to wait for unit-test by adding needs.job and then for it to download and use the artifact produced by unit-test by adding needs.artifacts.

unit-test:
  stage: analyze
  image: golang:1.18
  script:
    - go test -v -coverprofile=coverage.out -covermode=set ./...
  artifacts:
     paths:
       - coverage.out

sonar-analysis:
   stage: analyze
   image: sonarsource/sonar-scanner-cli:latest
   script:
     - sonar-scanner
   needs:
     - job: unit-test
       artifacts: true

You can add other configurations too, such as allow_failure: true so any failures won't block the rest of the stage. You can also specify the branches to run these by adding only.

   ...
   allow_failure: true
   only: 
     - develop
     - master
  ...

Also an important thing to note, this example does not cover everything. For example, you'll have to connect the sonar-analysis into your SonarQube dashboard, which we don't cover in this article.

4. Testing Gitlab CI jobs in local environment

Testing .gitlab-ci.yml changes was the trickier part, since it runs on Gitlab Runner and can't be tested with regular docker commands. In order to do this, we need to install Gitlab Runner on our machine, and then test the .gitlab-ci.yml file on top of it.

Gitlab Runner Installation

Install gitlab runner here, or install from linux repository

Setting up Gitlab Runner

Make sure installation succeeded by running gitlab-runner help
Register gitlab-runner

sudo gitlab-runner register

You will be prompted to input gitlab instance url and token, which can be found the Settings of your repo under CI/CD mennu. Expand Runners section and find the token under Specific runners

Enter runner description and tags
Choose executor, we'll use docker for our project. See here for other options
For detailed instruction, see here

Run test

After successfully setting up runner, you can start the service and then go to the project's root directory. To test one specific job, you can run

gitlab-runner exec docker job_name

Do this to test the whole pipeline

gitlab-runner run

If your gitlab-ci.yml isn't detected, a config flag can be added to specify file's location

gitlab-runner run --config /path/to/project/.gitlab-ci.yml

That's it! You should be able to execute Gitlab CI now on your machine and see SonarQube analysis running in real time.

Image by vectorjuice on Freepik

DEV Community: atikahe