DEV Community: Travis CI

Travis CI Pipelines: 2 Approaches to Source Control Branching

Montana Mendy — Wed, 17 Feb 2021 13:22:23 +0000

Feature branching is a game-changing aspect of modern software development. Being able to have a developer implement a new feature in a body of code in a safe, independent, isolated manner using Git branching is an overall positive approach to the way companies make software.

Over the years, two techniques of feature branching have emerged. One is what I call branching from repo. The other is branching from fork. Let’s explore each technique, as well as their benefits and tradeoffs.

Branching from repo

Branching from repo is a pattern in which feature branches are created off an existing branch in a business’s repository and then assigned to or assumed by a particular developer for implementation.

Figure 1: The branching from repo pattern is used to implement feature branches created from a particular branch in a business’s repository.

The branching from repo pattern is confined to creating feature branches that are within the boundaries of a particular business’s repository. Thus, only personnel or services that have the proper permissions for working that repository can create a particular feature branch.

The important thing to understand about branching from repo is that all the work that goes with creating the feature branch, developing the feature’s code, and then issuing the pull request for the completed code is done from within the boundary of the given repository. Those people on the “outside” have no ability to contribute changes.

Branching from fork

In the branching from fork pattern, a developer interested in adding to and improving upon code in a given repository forks the code into their own repository. Then, the developer creates branches against the forked code.

Figure 2: Developers use the branching from fork pattern to fork code from the business’s repository into the developer’s external repository to then create a feature branch.

Once work on a given branch is complete, the developer makes a pull request to merge the code back into the originating repository.

Figure 3: GitHub supports pull requests for merging code between repositories in separate accounts.

The important thing to understand about the branching from fork pattern is that it allows any developer to make improvements to code in an external repository in a controlled manner. All that’s required is view access to a repository of interest and then the cloning of the code.

Once the code is cloned, the developer is free to work on improvements according to their own software development process. The original repository is independent of the forked code; the only intersection between the two is when the pull request is made from the forked repository back to the originating repository.

Advantages and tradeoffs

Both patterns have advantages and trade-offs. Let’s examine them.

Branching from repo is well suited to private governance

Branching from repo is well suited for companies that want to keep their software development process private and subject only to their own development and project management practices.

Typically, a feature branch is created to accommodate a request that exists in a company’s backlog of work. In some cases, a feature branch can be created ad hoc, but usually, they are created by a governing authority within the company, such as a project manager or tech lead. Either the feature branch is created directly or as an ancillary action taken by automation within a project management platform such as Jira. As such, branching from repo allows project managers and tech leads to exert a lot of control over who can work on code. Also, the governing authority can determine the status of the feature branch by viewing the code as well as the branch’s commit history.

The trade-off is that branching from repo requires that those working on the code have a known relationship to the repository from within the source control management service hosting the code. For example, if the code is hosted on GitHub, a developer must have permission to make commits to the feature branch. Typically, this means inviting the developer to be a direct contributor to the repository or a member of an organization associated with the project. While this degree of formality provides control, it also excludes the general public from working on the code in an ad hoc manner, so there is less opportunity to benefit from the informal interest of others. Fortunately, branching from fork addresses this shortcoming.

Branching from fork is well suited to open-source development

Branching from fork is well suited for volunteer-based open-source development. Branching from repo is pretty much a “We’re assigning you some code to work” approach. On the other hand, branching from fork says, “Here’s some code. Feel free to work on it, but you need to figure out what you want to work on to make it better.”

In order for a developer to improve code using branching from fork, they need to identify possible improvements. This can take many forms. For instance, a developer can find problems to solve by going over the list of open issues in the source control management service that’s hosting the code. Once an issue is identified, the developer can address it according to their own development style and timeframe.

In addition to addressing known issues, a developer can implement, in an unsolicited manner, a feature they would like to see in the code. There’s no need to ask for permission. All that needs to be done is to implement the feature and then submit a pull request back to the original project. At that point, it’s up to the project’s maintainers to use the improvement. If the unsolicited feature is not accepted, it still exists in the fork and is available to interested parties there.

One drawback of branching by fork is that making contributions can require a good deal of work. Regardless of whether the developer is working on a known issue, implementing an existing feature request, or offering an unsolicited improvement, they will still need to learn the code. That learning curve can be significant, particularly if the developer has limited access to others who are willing to help. Sometimes the help required to fill in the gaps might be available, but many times it’s not. If the documentation is poor or the code is cryptic, figuring out how to make a meaningful improvement can take a lot of time. As those who have worked with volunteers can attest, the best volunteer experience is the one that’s the easiest. The more difficult the undertaking, the greater the risk of losing the volunteer’s sense of investment.

Conversely, a developer working on a feature request according to the branching from repo pattern will usually be a member of the business or organization that owns the repository. Thus, they will usually enjoy the support of other members who have experience working with the code. There’s both formal and tribal knowledge that’s available, so things are easier.

Another drawback of branching from fork is that since anybody can make a contribution, anybody will. This means that the code submitted via a pull request will vary in quality: Some of it will be quite good, but some of it will be quite bad. This creates more work for those who actually have to review pull requests in order to make sure that a submission meets quality standards. While a certain degree of quality control can be accomplished using automation, there are still aspects of the pull request that will require human inspection, particularly if the code is complex.

In short, branching from fork allows more people to work on code in their own manner, but making meaningful contributions is harder for all parties involved.

Putting it all together

The beauty of being able to branch source code is that it allows developers to work on code in a safe, controllable manner. Modern software development depends on branching.

The beauty of forking is that it provides a way for independent parties to work on an entire repository hosted in a source control management platform, such as GitHub or BitBucket, in an independent manner without affecting the original repository. Combining branching and forking creates a new dimension in software development.

Whether you use the branching from repo technique or the branching from fork technique, the important thing is that both methods provide developers a significant degree of independence while ensuring that the code that eventually makes its way into production will be the highest quality possible.

Configuring Travis CI to Run a Deno Project

Montana Mendy — Wed, 17 Feb 2021 12:52:42 +0000

Nothing lasts forever in the world of ephemeral computing. It’s the nature of the beast. Today, more companies are maximizing their IT budgets by practicing the principles of infrastructure as code (IaC). They’re creating and destroying virtual assets on demand in order to meet the needs of the moment.

Getting started

I am a big fan of Deno. I am also a big fan of Travis CI. What’s not to like? Deno is a powerful new programming framework that picks up where Node.js left off. Travis CI is a CI/CD platform that integrates easily with the projects stored in my GitHub repo. They’re both very cool.

But there’s a problem: Travis CI does not support Deno out of the box. Let's see what we can do.

Deno is still in its infancy, so it makes sense that it is not quite yet on the Travis CI radar. Still, no biggie. Travis CI is flexible enough to allow me to build support for Deno right into the CI/CD workflow that kicks off when my Deno project runs on Travis CI. In fact, getting Deno to run on Travis CI was pretty easy, and the results are very useful, so allow me to share.

First I’m going to cover the overall process for deploying to Travis CI and running a workflow defined in a .travis.yml file. Then I’ll show you the custom .travis.yml file I wrote that has Travis CI install Deno and run the unit tests for my Deno project.

Travis CI 101

Travis CI runs a CI/CD process right out of the box as part of its signup process. The first time you come to travic-ci.org, you’ll be asked to log in to Travis CI using your GitHub or BitBucket credentials. (The Bitbucket signup is in beta.)

The signup process asks you to declare repositories on the source code management (SCM) service, into which a webhook will be installed. Once a webhook is installed on the given repository, Travis CI will receive notifications about events that happen in that repository. For example, when a developer commits code into a repository that has a webhook installed, Travis CI will be notified about the commit.

Travis CI will clone the contents of that repo into a virtual machine instance running in the Travis CI domain. Once the runner VM has the contents of the cloned repository, it will look for a special file named travis.yml that contains instructions for provisioning the VM and about tasks to perform once the runner VM is provisioned. Figure 1 below describes the process.

Figure 1: The Travis CI Workflow

As you can see in callout 1 above, the first step in the workflow process is that a developer commits updated code to the repository of interest in the SCM service running in the cloud. A webhook in the repository “sees” the commit and emits an event message that is captured by Travis CI, as shown at callout 2. Travis CI clones the repository’s code into a VM in its domain and then looks for the .travis.yml file in the cloned files. The .travis.yml file is identified, and the configuration and instructions in .travis.yml are executed, as shown at callout 3.

The process is straightforward, and it’s quite powerful in that we can construct the .travis.yml file to define custom configuration and execute tasks that are special, too. In fact, we’re going to use travis.yml to install the Deno language on the runner VM and then execute the unit tests that are part of my Deno project. This is very cool because, remember, as of this writing Travis CI does not support Deno out of the box.

Let’s take a look at how support for Deno is implemented.

Getting Travis CI to support Deno

As mentioned above, the way you get Travis CI to do what you want it to do in its CI/CD process is to construct a .travis.yml file according to the format defined in the Travis CI build config specification.

The way I got Travis CI to support Deno was to do a bit of a hack in my project’s .travis.yml file. The entire contents of .travis.yml are shown below in listing 1. Let’s take a look at the details on a line-by-line basis:

language: default
os: ubuntu
services:
    - docker
branches:
    only:
    - master
before_install:
    - pwd
    - curl -fsSL https://deno.land/x/install/install.sh | sh
    - ls -l $HOME/.deno
    - export DENO_INSTALL="$HOME/.deno"
    - export PATH="$DENO_INSTALL/bin:$PATH"
    - deno run https://deno.land/std/examples/welcome.ts   
script: 
    - cd ./simplecalc/test/
    - sh run_test.sh
    - cd ../../fortune_cookies/test
    - sh run_test.sh

Listing 1: .travis.yml

Travis CI allows you to define a language that gets implemented in the VM runner. It supports a variety of languages out of the box, but sadly, Deno isn’t one of them. Thus, at Line 1 in the travis.yml file above, I set the value of the language attribute to default. It really doesn’t matter which language I define, because I am going to install Deno anyway.

At line 2 at the os attribute, I set the runner VM’s operating system to Ubuntu. Travis CI supports the macOS and Windows operating system too, but I use Ubuntu because my code is intended to run on Linux.

Travis CI allows you to configure the runner VM with a variety of services. For example, I could apply values to the services attribute that will install MySQL, Redis and Memcached to the runner VM automatically. In my case, I need to have Docker installed because one of my unit tests requires that a web server programmed under Deno be up and running as a Docker container. So at Lines 3 and 4, I set docker as a service.

Lines 5 through 7 set the branches attribute to tell Travis CI to use the code in the master branch of the cloned repository. You can set Travis CI to process specific branches in a given repo, but in this case, the code I’m interested in unit testing is in the master branch.

Lines 8 through 14 are where Deno support is implemented. I add the command line tasks required to download Deno. Once downloaded, the task will install Deno and then check that Deno is running. This is all defined under the before_install attribute, which indicates the stage before installation in the Travis CI job lifecycle. The install stage is the time when all dependencies are installed, so in this case, I am telling Travis CI to install Deno before any other download and configuration happens.

Line 9 tells Travis CI to output the present working directory (pwd). I do this to provide debugging information that might be useful later.

Line 10 runs the curl command that downloads and executes the shell (sh) script that does the actual work of installing Deno.

Line 11 lists the contents of the .deno directory. I do this to provide debugging information. When the Deno installation is run, it creates a .deno directory in the user’s $HOME directory where the Deno binaries and dependencies are stored.

Line 12 creates an environment variable, DENO_INSTALL which describes the location of the Deno binary and dependencies.

Line 14 runs the simple “Hello World” Deno application, which is downloaded directly from the Deno home site. I do this just to confirm that Deno is up and running properly.

Lines 15 through19 run the unit tests for the various subjects that are part of my main projects. (See figure 2, below.)

Figure 2: An excerpt from the console output created with travis.yml running my project’s unit tests

That’s it. As you can see, I got Deno up and running under Travis CI using a trivial amount of code in the .travis.yml file. Being able to do so much with such a small amount of code attests to the power that Travis CI brings to the CI/CD experience.

You can view the code for this Deno project on GitHub here.

Join the initiative

Travis CI provides the power and flexibility to make just about any CI/CD workflow possible. The platform’s versatility is reason enough for enterprises to consider adopting it. But when it comes to running Deno under Travis CI, there’s more!

Travis CI supports implementing community-based languages. Thus, it’s entirely possible to get Deno incorporated into the array of languages that Travis CI supports once the infrastructure is in place. All that’s required is for three members of the Deno community to commit to becoming maintainers and keeping the language support active. That being said, I am actively looking for people who want to become maintainers.

Making Deno work out of the box with Travis CI is a win-win for all parties. If you’re interested, contact me on LinkedIn. It will be a great adventure, and the result of our endeavor will benefit many for years to come.

The Cookbook: Build Matrix

Montana Mendy — Thu, 11 Feb 2021 10:53:30 +0000

A build matrix in Travis is made up by several multiple jobs that run in parallel. This can be useful in many cases, but the two primary reasons we see people use matricies is for reducing the overall build execution time and running tests against different versions of runtimes or dependencies to get the best version of the build. Let's learn about Build Matricies in Travis CI.

Build Matrix and setting up the .travis.yml

As you know by now, you need to tell Travis which language environment to select for your project. You can do so using the language key option, in this Cookbook we'll be using PHP. You can specify which PHP versions will the tests be executed. Not introducing patch versions tells Travis to use the latest available, which is sometimes referred to edge, or if it's the stable version, you guessed it, it's called stable. So let's start building out our .travis.yml:

language: php

php:
  - 5.3.3
  - 5.4
  - 5.5
  - 5.6
  - hhvm

Env Vars

Let's tell Travis to do multiple runs/builds with different sets of env var values. To do so, add an env key. Each segment is understood as a different environment and tests are run separately as such. We are going to use PHP_SEGMENT_TEST later in the file to run tester with the argument of -p and hhvm option in HHVM environment, so in your .travis.yml file, add the env segment:

env:
  - PHP_SEGMENT_TESTRUN="php-cgi"
  - PHP_SEGMENT_TEST="hhvm"

Now simple math would have it, the combination of five PHP versions and the two env vars generates a total of 10 runs.

Dependency installation using Composer

You'll obviously have an install segment in your .travis.yml. Each sub-segment means one single command. Composer installs your dev dependencies by default. You should use --no-interaction so Composer doesn't ask questions Travis can't answer, and can continue the build.

You'll want the latest build, so in you .travis.yml make sure you run the update flag at before_install, so it would look like:

before_install:
  - composer self-update

install:
  - composer install --no-interaction

The Build Matrix

Depending on the configuration above (different segments and sub segments), a build matrix wil be generated. The matrix contains all combinations the environment settings. A single combination is called a job and is run separately. You can modify the matrix in matrix section.

If you want to exclude a job, use the exclude key. In our case, we don't want to use -p hhvm parameter for standard PHP versions and -p php-cgi for HHVM:

matrix:
  exclude:
    - php: 5.3.3
      env: PHP_SEGMENT_TEST="hhvm"

    - php: 5.4
      env: TPHP_SEGMENT_TEST="hhvm"

    - php: 5.5
      env: PHP_SEGMENT_TEST="hhvm"

    - php: 5.6
      env: PHP_SEGMENT_TEST="hhvm"

    - php: hhvm
      env: PHP_SEGMENT_TESTRUN="php-cgi"

To your pleasure, you can define jobs that are allowed to fail without causing the whole build to shown as failed. To do so, declare allow_failures. For our sake, we allow HHVM to fail, lets do this via:

matrix:
  allow_failures:
    - php: hhvm

I only specified the PHP version, not the environment variables. This means all jobs with HHVM version will be allowed to fail/retry (if that is an option you have in your .travis.yml. Anyway, nevermind the environment variables values. It works with exclude also.

Running the scripts/tests

Tests are run in script segment of the .travis.yml. Let's assume your tests are in scripts/ folder in your file tree and you provide your own php.ini in the same folder. Additionally, we can tell TESTER to display information about skipped tests with the -s flag and to use value of earlier declared TESTER_SEGMENT_TEST as PHP binary with the -p flag:

script:
  - ./vendor/bin/tester -p $TESTER_SEGMENT_TEST -s -c ./tests/php.ini ./tests

If the build fails we want to use after_failure to get the exact values via:

after_failure:
  # Prints *.actual files contents (if not revert) 
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

Other services

Travis comes with multiple popular services and has no shortages of them, (e.g. MySQL, Redis, Docker) pre-installed. However, if you need to use for example Redis storage, you can tell Travis in services section via:

 services:
  - redis-server

DB Init

Depending on what database you chose to use, it can vary -- in this example we are using MySQL. MySQL runs on 127.0.0.1 and you can log in using travis or root as username. Say you have a DB setup script, and it's in tests/montana/testbase.sql., here's a sample segment:

 before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/montana/testbase.sql

.travis.yml in it's final form

After a lot of this segment config, this is what you're left with:

language: php

php:
  - 5.3.3
  - 5.4
  - 5.5
  - 5.6
  - hhvm

env:
  - TESTER_SEGMENT_TEST="php-cgi"
  - TESTER_SEGMENT_TESTRUN="hhvm"

matrix:
  allow_failures:
    - php: hhvm

  exclude:
    - php: 5.3.3
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: 5.4
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: 5.5
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: 5.6
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: hhvm
      env: TESTER_SEGMENT_TESTRUN="php-cgi"

services:
  - redis-server

before_install:
  - composer self-update

install:
  - composer install --no-interaction --prefer-source

before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/montana/testbase.sql

script:
  - ./vendor/bin/tester -p $TESTER_SEGMENT_TEST -c ./tests/php.ini -s ./tests/

after_failure: # to find actual variables if needed, a verbose esque way of doing things
  # Prints *.actual files content
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

Activate webhook

Go to www.travis-ci.com, and flip switch to ON for all repositories you'd like to enable. Travis will now add your repository to queue after every pushed commit or created pull-request. After a short while, your repository will be tested. Now if you'd like the opposite, there's a bash script I've created, which I will attach right below this - as some people find this more efficient:

#!/usr/bin/env sh

echo "Hello from Montana at Travis"
echo $TRAVIS_PULL_REQUEST

if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then
  surge --project ./dist --domain auto-deploy-test.surge.sh
else
  echo "This is a PR, not deploying"
fi

To know if your build is successful either check the GitHub status, or generate a status image.

Congratulations

This is Travis's Build Matrix in a nutshell, it can seem a bit confusing but after going over it a few times, it will start to make sense.

The Cookbook: Jekyll

Montana Mendy — Mon, 25 Jan 2021 08:24:54 +0000

GitHub Pages are an amazing way to host Jekyll pages, but in some cases, you might be interested in running your Jekyll page on a different host (like Azure Web Apps, Heroku, AWS). In this Cookbook let's setup a DevOps Build Pipeline that takes commits, run smoke screen testing, and actually deploying.

Getting started with Jekyll

Ideally with Jekyll, you want a compile script (obviously in bash) that traverses to an out/ directory. For sake of this example, you can call the bash script anything but I'm going to call it jekyll.sh. Your project may require something different, for example npm build or something to that extent.

When creating the file tree/structure, let's make sure the out/ directory contains almost or essentially everything you want deployed to gh-pages. In a case like this, you will 95% of the time have a index.html file.

Check this script into your project. Now this is just an example, now let's say the script just reads (remember, your script might just read npm build:

Using Bash


#!/bin/bash/env sh 

# only let the script proceed when started not by a pull request (PR)

if [ $TRAVIS_PULL_REQUEST == "true" ]; then # you can make this more strict via "==="
  echo "this is a PR, exiting now"
  exit 0
fi

# enable error reporting to the console & log that into a file entitled "log.txt" 

set -e

# build site with jekyll, by default to `_site' folder

bundle exec jekyll build

# find ./_site -name "*.html" -exec bundle exec htmlbeautifier {} \;

bundle exec htmlproof ./_site --disable-external --check-html --verbose

# cleanup using `rm -rf`

rm -rf ../Montana.github.io.master

Travis and adding Travis to your project

We assume you have one, but if not get a Travis account at https://travis-ci.com/. Turn on Travis for the repo in question, using the Travis GUI.

Encrypted credentials

If you read the Cookbook Series: Encryption, you can most likely skip over this, but if you haven't - when you deploy using Travis, ideally you want to deploy gh-pages without checking in the necessary credentials to your repo, this can be done using encryption.

Next up, you'll want to generate a SSH key. The advice I always give is to never reuse SSH keys. To generate a new SSH key, you'll want to open a terminal and run:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com" # in my case "ssh-keygen -t rsa -b 4096 -C "montana@travis-ci.org"

You'll be prompted to do a few things, one thing you want to remember is you do not want to include a password, just press enter when prompted, then follow the next prompt and continue. You'll now want to add the deploy key to your repo, and remember do not reuse SSH keys. You can do this via:

https://github.com/<github_username>/<your_repo>/settings/keys # add the deploy key to your repo

Now it's time to encrypt the key that was generated using the Travis CLI client. So now let's demonstrate on how we would add your deploy key to your specified repo.

Adding your encrypted vars

First, let's open an editor so we can view the dpl key, so let's run:

cat github_deploy_key.pub

You can also use touch, so instead of cat, it would be:

touch github_deploy_key.pub

Now that you have the key, make sure the proper repo is conditioned:

https://github.com/<github_username>/<your_repo>/settings/keys # add the deploy key to your repo

Now let's use Travis again from the CLI again to login:

travis login --org --auto

You'll then want to run:

travis encrypt-file 'github_deploy_key'

Make sure you add your PUBLIC key as well:

git add 'github_deploy_key.enc'

The bash script I've coded below automates the above process essentially, I did this to make it a bit easier on the Travis CI user, for the sake of time lets say this bash script I created is called dpl_key:

#!/bin/bash/env sh 

set -e

if [ -z "$GITHUB_TOKEN" ]; then
  echo ""
  echo "GITHUB_TOKEN environment variable is missing"
  echo ""
  echo "Travis dpl_key generator" 
  echo ""
  echo "This generates a public/private dpl_key, add the public key as a deploy key
  with write access to the origin remote github repo, encrypt the private key as
  github_deploy_key.enc and add the configuration necessary to use it in your .travis.yml file"
  echo ""
  echo "Give it a shot:"
  echo ""
  echo "  GITHUB_TOKEN=\`cat ~/secret/GITHUB_TOKEN\` ./generate_travis_deploy_key"
  echo ""
  echo "where ~/secret/GITHUB_TOKEN is a file containing a github token with write access to the current repository : (origin)"
  echo ""
  echo "You must have the Travis executable installed on your system and available in the PATH"
  echo ""
  exit
fi

url="$(git config --get remote.origin.url)"
reponame="$(echo $url | cut -d/ -f2 | cut -d. -f1)"

# generate a new private and public key

ssh-keygen -t rsa -b 4096 -f github_deploy_key -N '' -C $url -q 1>/dev/null

pubkey="$(cat github_deploy_key.pub)"

# add the PUBLIC key to the github repository as a deploy key with write access

curl https://github.com/<github_username>/<your_repo>/settings/keys -H "Authorization: token $GITHUB_TOKEN" --data @- << EOF
{
  "title": "travis deploy key",
  "key": "$pubkey",
  "read_only": false
}
EOF

# use travis to encrypt the private key as github_deploy_key.enc and remove the private key

travis encrypt-file github_deploy_key --add --no-interactive -w /tmp/github_deploy_key -f --pro
git add github_deploy_key.enc

# cleaning

rm github_deploy_key
rm github_deploy_key.pub

# bash script by montana mendy

.travis.yml

So, this is a .travis.yml file I've created, as you can see if you use my config it will automate the deployment for you, if you'd like to deploy manually, then just remove the line:

language: generic # this doesn't install any environment at all 

before_install:
- bash chmod +x ./dpl_key.sh # will give proper permissions via chmod to both scripts 
- bash chmod +x ./jekyll.sh

branches:
  only:
  - master

before_deploy:
- openssl aes-256-cbc -K $encrypted_0a644212b3ae3_key -iv $encrypted_0a644212b3ae3_key -in deploy_key.enc -out deploy_key -d


deploy:
  provider: pages
  local_dir: out
  deploy_key: deploy_key
  edge:
    branch: master

Now you should have 3 files, the ones we have talked about, jekyll.sh, deploy_key.enc and your .travis.yml. Make sure you've logged into Travis, let Travis know about the repo via syncing, once you push to GitHub it will compile and deploy your source, either using my bash script or doing it step by step.

Conclusion

I hope you learned some niche ways on deploying source while using Travis, until next time!

The Build, Test, Nuke Pattern

Montana Mendy — Thu, 21 Jan 2021 03:22:46 +0000

The Build, Test, Nuke Pattern

A Tutorial on Testing Etiquette in the Age of Ephemeral Computing

Paying only for what you use makes sense technically and financially. IaC offers a lot of benefits, but it does take some getting used to, particularly with regard to resource management.

There’s a certain etiquette involved when a lot of people are using a common set of resources. In a way, being part of an IaC environment is similar to using the laundry room in an apartment building. You need to be aware that other people are using the washers, dryers and folding tables. Thus, you need to keep moving your clothes along from washer to dryer in a timely manner, and you need to clean up after yourself, always.

The same is true in ephemeral computing. There is nothing more irritating than testing on a virtual machine that is all clogged up with useless applications and components left behind from previous tests. Not only do such remnants take up valuable computing resources, but they might also compromise the outcome of your testing.

When running tests in an ephemeral, IaC environment, a good pattern to follow is one that I call the Build, Test, Nuke pattern. Let’s explore the pattern using an example from a set of unit tests I created in my Deno demonstration project and ran on Travis CI.

Understanding Build, Test, Nuke

The structure of the Build, Test, Nuke pattern is to divide the testing event into three stages. In the first stage, Build, your testing script gets the source code from a source control management service, such as GitHub, and builds the required dependencies into a deployment unit, such as a Docker container. In the second stage, Test, the script tests the artifact. Then, in the third stage, Nuke, your script destroys the deployment artifacts used in the testing, removing them from both memory and disk. (See figure 1 below.)

Figure 1: The Build, Test, Nuke pattern

The benefit of the pattern is apparent. Your scripts are using only the resources needed, when they’re needed. For a small application, using the pattern might seem trivial. But, when you get into an application that has a dozen dependencies running in separate processes that put pressure on both memory and network IO, leaving them up and running after testing is done can be a significant risk. Destroying everything when testing is over addresses this risk head-on.

Now that we’ve covered the concept behind the Build, Test, and Nuke pattern, let’s take a look at how it’s implemented when running a set of unit tests under Travis CI.

Putting Build, Nuke, Test into Action

As mentioned above, I use the Build, Nuke and Test pattern in a set of HTTP tests I created for an application in my Deno demonstration project, which you can find [here].(https://github.com/reselbob/denodemo The application is called SimpleCalc. SimpleCalc is an API that exposes the math operations add, subtract, multiply and divide. The API runs as a web application over HTTP.

What is Deno?

Deno is a next-generation programming framework developed by Ryan Dahl, the creator of Node.js. Deno runs using the V8 JavaScript, as does Node.js, but Deno uses the Typescript language to program. Deno also improves upon Node.js in other ways, such as the way dependencies are stored and runtime permissions are granted. For more information about Deno, visit the website at: https://deno.land/.

The way the HTTP test works is the script creates the web server that hosts the SimpleCalc API in a Docker container. Then, once the SimpleCalc API server is up and running, a set of Deno tests are run. After the tests are run, the test script destroys both the container in which the SimpleCalc API server is running and the container image that’s used to create the container for the SimpleCalc API. (You can think of a container image as the template that describes the container that will run in memory.)

Listing 1 below shows the Bash script that creates the container, runs the HTTP tests, and then destroys the container and container image.

#!/bin/bash
cd ..

docker build -t mydenoserver .
docker run -d --name simplecalc -p 7700:7700 mydenoserver

deno test  --allow-net --allow-env --allow-read --allow-write --reload

docker rm -f simplecalc
docker rmi -f mydenoserver

Listing 1: The Bash script that implements the Build, Test and Nuke pattern

Notice that the Build phase takes place at lines 4 and 5. The script uses the Docker build command to create a container image named mydenoserver using the default Dockerfile stored along with the source code. (See line 4 above.) Then, at line 5, the script executes docker run to run a container named simplecalc using the container image mydenoserver.

Line 7 in listing 1 above invokes the unit tests that are stored in the test directory of the SimpleCalc app source code files. (The Deno test looks for test files in a directory named test by default.)

Listing 2 below shows an excerpt from the test file api_tests.ts. Notice at line 9 that the test accesses the SimpleCalc API’s add operation by way of the webserver running at http://0.0.0.0:7700/sum/8,4,3. The IP address 0.0.0.0 indicates the source address for the local computer on which the test is running. The SimpleCalc API is listening for traffic on port 7700. The /sum/8,4.3 segment of the URL is special to the SimpleCalc API. It describes the sum operation and an array of integers that will be summed up by the operation.

import { assertEquals } from "https://deno.land/std/testing/asserts.ts";

// The test, Can add from API, is an excerpt from a set of many tests
Deno.test({
    name: "Can add from API",
    ignore: false,
    async fn() {
        const url = `http://0.0.0.0:7700/sum/8,4,3`
        await fetch(url)
        .then((response) => {
            return response.text()
        })
        .then(data => {
            assertEquals(Number(data), 15);
        })
    },
  });

Listing 2: An excerpt from the test of HTTP tests that exercise the SimpleCalc API.

The important thing to know about listing 2 above is that it describes a test that is run against a container that hosts the web server, and that container was created on demand as part of the testing process.
After the test is completed, the Bash script controlling the entire process will issue these two commands as shown below, as well as above in listing 1 at lines 9 and 10:

docker rm -f simplecalc
docker rmi -f mydenoserver

These are the commands that remove both the container and the container image from the host environment. These two lines fulfill the last part of the Build, Test, Nuke pattern. As you can see, the container and the container image are removed completely from the virtual machine host.

Listing 3 below shows the .travis.yml file that gets run when new or adjusted code is committed to the demonstration Deno project.

language: default
os: ubuntu
services:
    - docker
branches:
    only:
    - master
before_install:
    - pwd
    - curl -fsSL https://deno.land/x/install/install.sh | sh
    - ls -l $HOME/.deno
    - export DENO_INSTALL="$HOME/.deno"
    - export PATH="$DENO_INSTALL/bin:$PATH"
    - deno run https://deno.land/std/examples/welcome.ts

script: 
    - cd ./simplecalc/test/
    - sh run_test.sh

Listing 3: The Travis.yml file that installs Deno executes the script run_test.sh, which implements the Build, Test, Nuke pattern in the Travis runtime environment.

One of the nice features of Travis CI is that I can register the Deno demonstration project that’s hosted on GitHub with Travis. Once registered, Travis will install a webhook in the GitHub repo that makes it so when code is committed to the repo, the repo will be cloned automatically into a Travis CI virtual machine dedicated to my testing session. Travis looks for the .travis.yml in the source code and then, once identified, will configure the virtual machine according to settings in the .travis.yml file.

In this case, the instructions to install Deno are at lines 8 through 14 in listing 3 above. The instruction to navigate to the directory that has the Bash script that controls the testing process is at line 17. The instruction to execute the Bash script is at line 18.

Remember, the Bash script contains all the instructions to execute the Build, Test, and Nuke pattern. Mission accomplished!

Putting It All Together

The Build, Test, and Nuke pattern is simple and powerful. Using it consistently as part of your CI/CD process will help you run your tests as a reliable, good neighbor in the host runtime environment, whether they’re running on your local machine, under Travis or anywhere else in the cloud.

The Cookbook: Fable

Montana Mendy — Mon, 18 Jan 2021 11:03:44 +0000

Fable produces readable JavaScript code compatible with ES2015 standards and popular tooling like Webpack, which you've probably heard of if you've ever used React. Let's start this Fable.

Build Fable with Travis CI

Hey builders let's get to building Fable with Travis CI! Let's start out with your .travis.yml file:

language: minimal

services:
  - docker

sudo: required

before_install:
  - docker pull vbfox/fable-build

jobs:
  include:
    - stage: "CI"
      script: docker run -v "${PWD}:/app" -w "/app" vbfox/fable-build bash -c "yarn && yarn postinstall && yarn build"

There's sometimes issues with Fable for whatever reason with syntax errors, so you may want to take the time and lint your .travis.yml file, you can do this via running travis lint.

The `package.json`

The package.json contains something like the following:

    "scripts": {
    "start": "webpack-dev-server",
    "build": "webpack -p",
    "postinstall": "paket restore && paket generate-load-scripts -f netstandard2.0 -t fsx",
    "predeploy": "npm run build",
    "deploy": "gh-pages -d docs"
  },

As you can see webpack is being triggered there as well. That's it - you've now built Fable in Travis CI! Phew, that was easier than we thought.

The Cookbook: Branch Flow

Montana Mendy — Tue, 12 Jan 2021 03:29:35 +0000

It's important when using multiple branches in your repository to define which branches to build to, and which branches not to build to. Let's get started.

Branch Flow

Travis CI uses the .travis.yml file from your project structure, once you've configured your flow via the .travis.yml file, you can trigger a build by pushing changes through the CLI using:

git init
git add . 
git commit -m "Travis build branch" 
git remote add origin remote repository URL
git remote -v 
git push -u origin master

You can tell Travis to build multiple branches using blacklists or whitelists. More specifically you can define which branches to build using a whitelist, or on the flip side you can use blacklist. You can blacklist branches that you do not want to be built via the following:

# blacklist (branches you don't want to be built) 
branches:
  except:
    - legacy
    - edge

# whitelist
branches:
  only:
    - master
    - stable

If you specify both whitelist and blacklist, only takes precedence over except. By default, let's say you have a gh-pages branch, that branch is not built unless you add it to the whitelist explicitly. When deploying to your version control, or if you're using Apache SVN (Subversion), in some cases, it may reset your working directory, it does this via:

git stash --all

As a cursory cautionary move, you can add skip_cleanup to your .travis.yml file. So after, your yml file might look like this:

deploy:
  skip_cleanup: true

If there's anything you want to add before you deploy, you can obviously add before_deploy. This is also true for an inverse method, for after deployment via after_deploy.

Deploying and Using WebAssembly Under Deno on the Server Side Using Travis CI

Montana Mendy — Thu, 07 Jan 2021 11:05:23 +0000

Containers are a game changer in software development. They provide the operational isolation found in virtual machines without the overhead. Whereas it can take a virtual machine minutes to spin up, you can have a container up and running in seconds, or even in some cases milliseconds.

Getting started

Containers have become central to modern distributed application architecture, particularly now that container orchestration technologies such as Kubernetes and Docker Swarm have become commonplace in the enterprise. However, as popular as container technology is for implementing isolated processes for use in a distributed application, there is an alternative. That alternative is WebAssembly.

WebAssembly is a compiled binary that operates inside a runtime unit called a WebAssembly VM, or virtual machine. WebAssembly has been around for a while on the client side, and all the major web browsers support it. But you can also run it on the server side. All you need is to have a programming framework that supports server-side WebAssembly VM.

Deno — an emerging language from Ryan Stahl, the creator of Node.js — supports WebAssembly. Deno uses the V8 JavaScript runtime found in Chrome on the server side, as does Node.js. The WebAssembly VM is built right into V8, so Deno supports WebAssembly.

In this tutorial, I am going to show how to use Travis CI to build a WebAssembly binary written in Rust, and then run a demonstration application I wrote in Deno that uses the WebAssembly as an internal component.

This tutorial is high-level, and you don’t need to know the details of Rust or Deno to get benefit from reading it. The hope is that the tutorial will spur your interest to delve further into the technologies.

Understanding the demonstration application

The demonstration project that accompanies this article is a Deno application that uses a WebAssembly binary. The Deno code uses logic in the WebAssembly to do a calculation. The calculation that the WebAssembly binary provides is encapsulated into a function named cube(). The function cubes a number that is provided as a parameter. For example, when the number 3 is passed to the function, as cube(3), the function returns 27.

As shown in figure 1 below, the WebAssembly (WASM) binary is written in the programming language Rust. Rust is a compiled language. A programmer writes some code in text and stores it in a file named lib.rs. The code is passed to the Rust WASM compiler. The result is a WebAssembly binary that’s used by the Deno code.

Figure 1: Deno uses a WebAssembly binary programmed in Rust.

That’s the high-level view. Let’s take a look at some of the detail involved in creating the WebAssembly binary in Rust and then consuming the binary in Deno.

Examining the WebAssembly code

The work required to create a WebAssembly binary in Rust is consolidated using Rust’s crate packaging technology.

The Rust programming community provides a packaging ecosystem similar to npm for Node.js and to Java’s Maven. Packages under Rust are called crates. The standard file that describes a crate is named Cargo.toml. It’s similar to the way Node.js uses package.json to describe a Node.js package. (I’ll explain the Cargo.toml file for the demonstration Rust binary in a moment.)

The way you install Rust packages from an external Rust package repository is to use the command cargo install .

Creating a WebAssembly in Rust can be a detail-laden undertaking. To simplify the process, we’re going to use an external package called wasm-pack. wasm-pack will execute against the Cargo.toml file that defines the WebAssembly binary we’re going to create. (Keep in mind that the standalone WebAssembly binary is considered a package.)

Listing 1 below shows the Cargo.toml file that describes the Rust project.

[package]
name = "cube_it"
version = "0.9.0"
edition = "2018"
authors = ["Bob Reselman <reselbob@gmail.com>"]
description = "A small project to demonstrate how to run to create a WebAssembly binary"
license = "MIT/Apache-2.0"
repository = "https://github.com/reselbob/deno-wasm"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wasm-bindgen = "0.2"

[lib]
crate-type = ["cdylib", "rlib"]

Listing 1: The Cargo.toml file

Lines 1 through 12 above provide typical metadata about the project. Notice in line 1 that the arbitrary name of the package is cube_it. This makes sense because the WebAssembly binary that this package definition creates has a function that will cube a number.

Lines 13 and 14 indicate that the Rust project will use an additional package, wasm-bindgen. The purpose of wasm-bindgen is to provide interoperability between the Rust binary and JavaScript code. Deno programming is done in TypeScript. However, at runtime TypeScript transpiles into JavaScript. Using wasm-bindgen allows the Rust code to access JavaScript functions such as logging and the JavaScript DOM model, so wasm-bindgen is defined in Cargo.toml.

Lines 16 and 17 assign the library cdylib to crate-type. This provides interoperability with C code. In addition, crate-type is assigned the value rlib to indicate that a Rust library will be produced.

Listing 2 below shows the Rust code for the function cube(x: u32). The function will reside in the file lib.rs.

#[no_mangle]
pub extern "C" fn cube(x: u32) -> u32 {
    x * x * x
}

Listing 2: The Rust code in lib.rs that publishes the cube function

The logic in the function cube(x: u32) will be consumed by the Deno code.

Once the Rust package is declared in Cargo.toml and the cube() function is defined in lib.rs, we’ll need to create the Deno code to consume the WebAssembly binary that’s created.

Running WebAssembly under Deno

Listing 3 below shows the contents of the file main.ts. This file contains the Deno code that consumes the WebAssembly binary that contains the function cube().

const wasmCode = await       Deno.readFile("./rust_for_deno/target/wasm32-unknown-unknown/release/cube_it.wasm");
const wasmModule = new WebAssembly.Module(wasmCode);
const wasmInstance = new WebAssembly.Instance(wasmModule);
const {cube} = wasmInstance.exports;

console.log(cube(1));
console.log(cube(2));
console.log(cube(3));
console.log(cube(4));

Listing 3: The Deno code that consumes the WebAssembly binary

Lines 1 through 5 do the work of loading the WebAssembly into the V8 virtual machine and making the cube() function accessible to the Deno code. Lines 7 through 10 execute the cube() function four times, each time with a different parameter value, and then write the output of each call to the console.

Now that we have both the Rust WebAssembly and Deno projects coded and ready to go, we need to create a travis.yml file that will build the WebAssembly binary, as well as install the Deno runtime environment and run the Deno code.

Working with the `.travis.yml` file

One of the nice features of Travis CI is that it will execute a run script in response to an event generated from a webhook installed in a GitHub repository that is bound to Travis CI. Under Travis CI, the run script has a standard name, .travis.yml.

The deno-wasm demonstration project for this article stored on GitHub has a webhook installed that sends an event notification to Travis CI whenever code is committed to the deno-wasm repository. Travis CI receives the event notification and clones the code into a runtime virtual machine created just for the project. Then, it runs the instructions in the expected .travis.yml file.

Listing 4 below shows the contents of the .travis.yml file.

language: rust
os: ubuntu
branches:
    only:
    - master
before_install:
    - pwd
    - rustc --version
    - sudo apt-get update
    - sudo apt-get -y install libssl-dev pkg-config 
    - rustup target add wasm32-unknown-unknown
    - cargo install wasm-pack
    - export PATH="$HOME/.cargo/bin:$PATH"
    - cd ./rust_for_deno
    - wasm-pack build
    - cd ..
    - curl -fsSL https://deno.land/x/install/install.sh | sh
    - ls -l $HOME/.deno
    - export DENO_INSTALL="$HOME/.deno"
    - export PATH="$DENO_INSTALL/bin:$PATH"
    - deno run https://deno.land/std/examples/welcome.ts

script: 
    - deno run --allow-read main.ts

Listing 4: The travis.yml file for building and running the deployment on Travis CI

The first thing to notice about the .travis.yml file in listing 4 above is that Travis CI supports the Rust programming language right out of the box. All you need to do is set the language definition, as shown at line 1.

The work of creating the WebAssembly binary and installing Deno is done in the before_install section, starting at line 5.

Lines 7 and 8 are convenience instructions that report the present working directory (pwd) and the current version of the Rust compiler installed (rustc --version). This is done just to make sure that the .travis.yml script is talking to the Travis CI runtime in a predictable manner.

Lines 9 and 10 install some added packages that Ubuntu needs in order to execute the wasm-pack instruction that will come.

Lines 11 through 15 do the work of installing the wasm-pack Rust package and creating the WebAssembly binary using wasm-pack.

Deno is not installed in the Travis CI runtime environment by default. Thus, lines 17 through 20 do the work of installing it and configuring the DENO_INSTALL environment variable. Also, the PATH environment variable is updated to accommodate the Deno binaries.

Line 21 runs a Deno Hello World program downloaded from the internet, to verify that Deno is up and running.

As mentioned earlier, the code that uses the cube() function in the WebAssembly binary is stored in the file main.ts. main.ts is called within the script section of the .travis.yml file at line 24 above.

The output of the call to the main.ts file is shown below in figure 2.

Figure 2: The output from the demonstration application running under Travis CI

So there you have it. We’ve just used the Travis CI runtime environment to create a WebAssembly binary and run it in a Deno program. Granted, this is just a small demonstration program, but it does reveal the critical aspects of using WebAssembly to provide services to interested consumers in an isolated manner.

Putting it all together

It’s interesting that Solomon Hykes, the co-founder of Docker, said this about WebAssembly:

"“If WASM+WASI [WebAssembly and its system interface] existed in 2008, we wouldn’t have needed to create Docker."

Server-side WebAssembly has significant potential. Its elegance and power make it an attractive technology for the fast execution of complex algorithms on the server side. Consequently, there are a growing number of projects that are publishing tools for WebAssembly development. You can already create WASM binaries in Go, Rust, C# and C/C++. In addition, there are initiatives to bring support for WebAssembly to a wider array of languages. For example, AssemblyScript is intended to allow developers to create WebAssembly binaries from JavaScript code.

The opportunities at hand are significant. Of course, in order to make WebAssembly a viable addition to your programming toolbox, you will need to take the time to master the details of the technology. But, as current trends reveal, WebAssembly is becoming a mainstay on the server side, and realizing its value will be an investment that will yield dramatic returns.

WebAssembly is here. The time to take advantage of the opportunities at hand is now.

The Cookbook with Bash

Montana Mendy — Mon, 21 Dec 2020 02:19:51 +0000

Many projects on GitHub use Travis to automatically execute certain scripts on every build. Among these many scripts, there is one that's definitely the most well known, it's called Bash. Automate your shell commands, add env vars, and increase your workflow, plus the endless of other things you can do with Bash given it's flexibility. At the crux -- Bash on more occasions than not are crucial to Travis CI builds.

Getting started with Bash (gh-pages example)

Some goals you may have if you're wanting to automate processes in Bash are automating GH_REF value in gpages_build.sh script with the TRAVIS_REPO_SLUG.

In order to make the GH_REF variable automatic and not have to use process load everytime, Travis CI gives us the TRAVIS_REPO_SLUG which you may have seen in languages like React. The SLUG variable, which is basically the username/repo for whatever GitHub repo you're working on. You write it out like this: GH_REF="github.com/${TRAVIS_REPO_SLUG}"

Below is an example script of just what I've explained above, we can assume this Bash script is entitled pages.sh:

# This script pushes a demo-friendly version of your element and its
# dependencies to gh-pages.

# usage gp Polymer core-item [branch]
# Run in a clean directory passing in a GitHub org and repo name

#!/bin/bash

GH_REF="github.com/${TRAVIS_REPO_SLUG}"

org=`echo ${TRAVIS_REPO_SLUG} | cut -f 1 -d /`
repo=`echo ${TRAVIS_REPO_SLUG} | cut -f 2 -d /`

name="Montana"
email="montana@travis-ci.org"
branch=${3:-"master"} # default to master, when branch isn't specified

mkdir temp && cd temp # make temp dir 

# make folder (same as input, no checking!)
mkdir $repo
git clone "https://${GH_TOKEN}@${GH_REF}" --single-branch # you can theoretically as Montana likes to do, 'git stash pop' here

# switch to gh-pages branch
pushd $repo >/dev/null
git checkout --orphan gh-pages

# remove all content
git rm -rf -q .

# use bower to install runtime deployment
bower cache clean $repo # ensure we're getting the latest from the desired branch.
git show ${branch}:bower.json > bower.json
echo "{
  \"directory\": \"components\"
}
" > .bowerrc
bower install
bower install $org/$repo#$branch
git checkout ${branch} -- demo
rm -rf components/$repo/demo
mv demo components/$repo/

# redirect by default to the component folder
echo "<META http-equiv="refresh" content=\"0;URL=components/$repo/\">" >index.html

git config user.name $name
git config user.email $email

# send it all to github
git add -A .
git commit -am 'Deploy to GitHub Pages'
git push --force --quiet -u "https://${GH_TOKEN}@${GH_REF}" gh-pages > /dev/null 2>&1

popd >/dev/null

Implementing this in your `.travis.yml`

So below is a travis.yml file I've created, and let's say you named your script pages.sh, your .travis.yml file would look something like this:

language: bash

sudo: required

script:
    - chmod u+x bash pages.sh

There you have it, you have now successfully used Bash in Travis CI. You can do something a little easier as well - just to test if you have the steps down, maybe make a Bash program just called test.sh and have it read:

#!/bin/bash

echo "Hello Travis"

Then have your .travis.yml file read the following:


language: bash

script: chmod u+x test.sh

You should then see when running the build, Travis printing out Hello Travis. You now know you're using Bash successfully within Travis CI.

The Cookbook: Getting started with R

Montana Mendy — Mon, 07 Dec 2020 10:43:22 +0000

Some of the most downloaded R packages are built on Travis CI. In this cookbook, you'll see how easy it just is to setup Travis CI in an existing R project.

Getting started with R

Let's assume you have a file called basic_operations.R, and it reads something like this:

add <- function(a, b){
  result <- a + b
  return(result)
}


subtract <- function(a, b){
  result <- a - b
  return(result)
}

divide <- function(numerator, denominator){
  if(denominator == 0){
    stop("Nah")
  }
  else {
    result <- numerator / denominator
    return(result)
  }
}

Implementing Travis CI

Now, you want to have automated testing. Thankfully there's Travis CI! Let's start off with the .travis.yml file:

language: R
cache: packages
warnings_are_errors: false

Unit Tests

For unit tests, we want to use something called testthat. Let's say we named our whole project sampleR. Your initial testthat.R file would look similar to this:

library(testthat)
library(sampleR)

test_check("sampleR")

In your tests directory, you'll notice /tests/testthat and let's say you have a file in there entitled test-basic_operations.R, and it goes as follows:

test_that("addition works", {

  a <- runif(1)
  b <- runif(1)

  expect_equal(a + b, add(a, b))

  a <- -1 * runif(1)
  b <- runif(1)

  expect_equal(a + b, add(a, b))
})

test_that("subtraction works", {

  a <- runif(1)
  b <- runif(1)

  expect_equal(a - b, subtract(a, b))

  a <- -1 * runif(1)
  b <- runif(1)

  expect_equal(a - b, subtract(a, b))
})

test_that("division errors if zero", {
  expect_error(divide(1, 0), "Nah")
})

test_that("division works", {

  a <- runif(1)
  b <- runif(1)

  expect_equal(a / b, divide(a, b))

  a <- -1 * runif(1)
  b <- runif(1)

  expect_equal(a / b, divide(a, b))
})

You can see in this unit test, we are testing if division, addition, and subtraction passes the unit test.

Finishing up

At this point, you're all set, and ready for your R project to build. Last but not least, there are alternative ways to get packages when building out your .travis.yml. For example if I wanted to get packages from GitHub, it would look something like this:

r_github_packages: r-lib/testthat

The other alternative is to use CRAN. This is how it would look grabbing your package from CRAN:

r_packages:
  - testthat

Slight variants, but just different methods on getting the same package. It just comes down to your own approach and method.

Anchore Policy Enforcement with Travis CI

Montana Mendy — Fri, 04 Dec 2020 14:16:00 +0000

Travis CI Pipelines: Anchore

The Anchore Engine is an open-source project that provides a centralized service for inspection, analysis, and certification of container images. The Anchore Engine is provided as a Docker container image that can be run standalone or within an orchestration platform such as Kubernetes, Docker Swarm, Rancher, Amazon ECS, and other container orchestration platforms.

Anchore Engine can be used in several ways:

Standalone or interactively.
As a service integrated with your CI/CD to bring security/compliance/best-practice enforcement to your build pipeline.
As a component integrated into existing container monitoring and control frameworks via integration with its RESTful API.
Policy enforcement.

Implementing Anchore into your project via the `.travis.yml` file to check your containers

First, let's take a look at your .travis.yml file:

---
sudo: required
dist: bionic
notifications:
  slack:
    on_failure: always
matrix:
  fast_finish: true
  include:

    # https://github.com/anchore/anchore-engine
    # https://docs.anchore.com/current/docs/engine/quickstart/
    - name: "Scan images w Anchore Engine w docker-compose wo snaps on bionic amd64"
      dist: bionic
      arch: amd64
      services:
        - docker      
      language: python

As you can see above, you're off to a good start, you have your dist, sudo specified, matrix specified with a job name. The job will be running bionic, the architecture will be amd64 and of course Anchore is going to be checking the Docker container via docker-compose, and the way this will happen is once Anchore is fetched and docker-compose ps is ran, it will show the status of the container. Let's finish the .travis.yml file. We'll want to start with the before_install hook that Travis offers.

 before_install:
        - pip3 install virtualenv
        - virtualenv -p $(which python3) ~venvpy3
        - source ~venvpy3/bin/activate
        - pip install -r requirements.txt
      script:
        #  Download the docker-compose.yaml file and start
        - curl https://docs.anchore.com/current/docs/engine/quickstart/docker-compose.yaml > docker-compose.yaml
        - docker-compose up -d
        # Verify service availability
        - docker-compose ps
      after_success:
        - deactivate

You'll see the script hook using cURL to fetch Anchore, then copying the docker-compose.yaml file. It then makes sure the container is up by running docker-compose up -d. Now you'll want to verify service, the .travis.yml file will run docker-compose ps. Then of course, after it's done it will deactivate itself. This is Anchore in a simple .travis.yml file. You should see the Anchore table with it's findings at the end of your build as follows:

Further setup

Travis allows docker command execution by default, which makes integrating Anchore Engine as simple as adding the inline_scan script to your existing image build pipeline.

services:
  - docker

env:
  - IMAGE_NAME="btodhunter/anchore-ci-demo" IMAGE_TAG="travisci"

script:
  - docker build -t "${IMAGE_NAME}:ci" .
  - curl -s https://ci-tools.anchore.io/inline_scan-v0.6.0 | bash -s -- "${IMAGE_NAME}:ci"
  - echo "$DOCKER_PASS" | docker login -u "$DOCKER_USER" --password-stdin
  - docker tag "${IMAGE_NAME}:ci" "${IMAGE_NAME}:${IMAGE_TAG}"
  - docker push "${IMAGE_NAME}:${IMAGE_TAG}"

Alternatively you can setup cURL to fetch Anchore like this:

- curl https://docs.anchore.com/current/docs/engine/quickstart/docker-compose.yaml > docker-compose.yaml

Anchore Policy Enforcement

It's fairly easy with Travis and Anchore to setup Anchore Policy Enforcement. You can setup policy enforcement with one command:

 curl -s https://ci-tools.anchore.io/inline_scan-v0.6.0 | bash -s -- [options] IMAGE_NAME(s)

These are the scan options Anchore offers:

-b  [optional] Path to local Anchore policy bundle.
-d  [optional] Path to local Dockerfile.
-v  [optional] Path to directory to be mounted as docker volume. All image archives in directory will be scanned.
-f  [optional] Exit script upon failed Anchore policy evaluation.
-p  [optional] Pull remote docker images.
-r  [optional] Generate analysis reports in your current working directory.
-t  [optional] Specify timeout for image scanning in seconds (defaults to 300s).

What you read above bodes well for you. With Anchore, you essentially can scan local images before pushing them into a registry, allowing you to inject scans as needed directly into your current workflows and enforce Anchore policy. You can see your policies being checked, look for the following:

If you look above, you can see the policies being enforced, scanned, and more. I recommend looking at the scan options that were laid out earlier for more verbose and customized options.

Why use security scanners for containers?

Running image scanners during continuous integration specifically scanning container images can be a cumbersome process that could be asynchronously executed during continuous integration (CI) using Travis. Including security scans in CI also allows developers to identify vulnerabilities quickly, before new software versions are released.

This is why policy enforcement is crucial. Anchore is perfect for your Travis CI builds.

A Short Journey into Source Control Branching and Release Patterns

Montana Mendy — Tue, 01 Dec 2020 13:54:28 +0000

Git is a transformational technology. It’s the foundation of most of the source code management (SCM) services used today. Git’s branching, forking and merge capabilities provide developers with the freedom to work with a high degree of independence while allowing companies to control the quality of the code that makes its way into production.

A Short Journey into Git Branching and Release Patterns

The process is a win-win situation for all parties involved in the software development lifecycle. A developer works within an SCM service such as GitHub, GitLab, Assembla or Bitbucket to create a branch or fork of a repository. Then, after work is completed, the developer submits a pull request within the SCM service. The code is inspected by the project’s experts. If the code passes the project’s quality standard, it’s accepted into the project. If not, the developer is notified about how to make improvements.

The process has become commonplace. Yet, for all the conformity that’s taken place around pull requests, one place where standardization is evolving is around the branching and deployment patterns a given company or project uses. There is no one set way of doing things; rather, there are a few different patterns in play. They're well worth taking a look at.

The following is an analysis of the branching and release patterns used by a few open-source projects. We’ll look at some large and well-known projects — Kubernetes, Ansible and Envoy — and the Deno programming language, which is growing in popularity but small when compared to something on the order of Kubernetes.
Understanding the different patterns is useful to those who already have or are planning to have open-source projects and need to determine a branching and release pattern to best suit their particular needs.

Kubernetes

Kubernetes is a major player in the technology landscape. It’s a container orchestration system that intends to run large, distributed applications in a fail-safe, ephemeral manner. Kubernetes is designed to run big applications in a big way.

Kubernetes uses a branching pattern in which all new features are created off the master branch. Work is done on the feature branch. Then, when the work is completed, a pull request is made. If approved, the code is merged from the feature branch into the master branch. When code in the master branch is determined to be ready for a version release, one of the project’s release managers will create a release branch from the master branch. (See figure 1 below.):

Figure 1: Kubernetes creates a branch for each major or minor release version.

Then, the release manager will use GitHub’s release feature to tag the release and create the deployment binaries. The deployment binaries are compressed into .zip and .tar.gz files.

What’s interesting about the Kubernetes approach is that creating a formal feature branch within the Kubernetes repository is a rare occurrence. Rather, a developer creates a fork of the source code from the Kubernetes repo and into their own repository. Then, when improvements in the forked code are completed, the developer creates a pull request to merge their code from the fork and into the main Kubernetes repo, as shown in figure 2 below:

Figure 2: Most feature improvements in the Kubernetes project are forks that become pull requests.

However, not all pull requests go up against the master branch. Sometimes pull requests are made against pre-existing version branches. The Kubernetes project has a very specific way of releasing versions of code. The master branch contains the code that will be put into the next release. It is the culmination of all work done over the project’s lifespan. When that master code is deemed ready for a new version release, a new version branch is created. That branch is named according to a release-x.y naming convention — for example, release-1.19. (See figure 1 above.)

As mentioned above, once the version branch is created, it’s made available to the public as a .zip or .tar.gz file using GitHub’s release feature. For example, the code for version branch release-1.19 was released as v1.19.0.

Developers can still make backport improvements to a specific version by making a pull request against that version’s branch. (Usually, these types of changes are bug fixes.) However, when it comes time to release code that improves an existing version branch, a new version branch is not created. Rather, a new release tag is created against the existing version branch. For example, when code improvements were made that were specific to v1.19.0 of Kubernetes, the next release was v.1.91.1. This release naming pattern is consistent with the guidelines defined by the semantic versioning convention.

You can read the Kubernetes release policy here.

Ansible

Ansible is an environment provisioning tool. Companies use it to create computing environments such as a virtual machine (VM), and then provision that VM with an operating system, network configuration and applications.

Ansible takes an approach that is somewhat similar to Kubernetes’ branching and release patterns but with a slight difference. In Ansible, the default branch is called devel; in Kubernetes, the default branch is named master. As with Kubernetes, Ansible will create a new branch for an intended new release. These branches are given the prefix stable-, as shown in figure 3 below:

Figure 3: The default branch in Ansible is called devel.

Developers who want to make a contribution to the project fork the Ansible repository into their own accounts. Then, they make improvements. Then, for the most part, pull requests are made against the devel branch. New features added to the devel branch will appear in the next version release.

Adding bug fixes is not as straightforward as adding new features. Sometimes a fix can be applied to devel and then directly applied to legacy release branches. But as Ansible reports, some bug fixes made to the devel branch might not cleanly rebase onto the given stable- branch. As a result, there are times that code from the devel branch needs to be changed a bit before it can be applied to a previous version release.

You can read Ansible’s release policy here.

When it comes time to releasing code, Ansible uses GitHub’s release feature. When a stable- branch is deemed ready for release, GitHub’s release feature is applied. This tags the stable- branch with a release label and packages the deployment code into .zip and tar.gz files. This is very similar to the process used by Kubernetes.

Envoy

Envoy is a popular L7 proxy and communication technology that facilitates interservice communication in distributed applications. It is an open-source project published under the sponsorship of the Cloud Native Computing Foundation.

Envoy uses a branching and release pattern that is nearly identical to Kubernetes and Ansible. As with Kubernetes, the default branch is named master. And, as with Kubernetes and Ansible, once code in the master branch is deemed ready for a version release, the maintainers of the Envoy project will create a branch for the particular release. They then use GitHub’s release feature to tag the release branch with the designated version number and create compressed binaries for the code for the release in .zip and .tar.gz formats.

In terms of supporting fixes for versions of code that have already been released, Envoy release management applies security fixes directly to the master branch and then onto the release branch, adding a release tag accordingly. Should a security patch be applicable to previous, supported versions, the maintainers of the legacy branch will be notified of the available fix. Each maintainer will apply the fix to the corresponding version branch they support and to an upgrade release of that legacy branch, as shown in figure 4 below:

Figure 4: Envoy creates release branches and then updates minor version releases against a particular release branch.

The technique used for creating feature branches is also similar to the technique used by Kubernetes and Ansible. For the most part, contributors make a fork of the Envoy repository and do work on their forks. When code is ready for contribution, the developer makes a pull request to apply the code in the fork to the master branch.

You can read Envoy’s release policy here.

This process is nearly identical in execution to the way Kubernetes and Ansible do branching and release management.

Deno

Deno is the next-generation development language started by Ryan Dahl, the creator of Node.js. It is an open-source project.

Deno developers make contributions to the project by forking the Deno repository into the developer’s account and then doing work on the fork. As you’ve seen, this is a very common pattern. However, unlike the large projects shown above, Deno does not create release branches from the master when it comes time to deploy code. Instead, Deno uses a simple branching and release pattern.

In Deno this is only one branch, master. (See figure 5 below.) There are no release branches as in the other projects described above:

Figure 5: Deno releases versions from a single master branch without creating release branches.

When the release managers at Deno decide a new version of code is deemed ready for release, they use GitHub’s release feature to tag the code according to a version release name, and then create the release’s .zip and .tag.gz binaries.

Deno does not document its release process. According to Kitson Kelly, a frequent contributor to the Deno project, “I don't think we will write it down, as it then becomes something more difficult to change.”

The simplicity of Deno’s branching and release pattern serves the project well. Deno is still a new, maturing project, and it has more flexibility in how it gets code out the door than a project that’s been around for awhile.

Also, Deno has a much more limited scope of activity compared to projects such as Kubernetes, Ansible and Envoy. Thus, its “release from trunk” pattern is sustainable. While the simplicity of Deno’s branching and release pattern is effective now, it will be interesting to see how Deno’s branching and release pattern evolves as the project grows.

Putting it all together

Creating and deploying software is a complex process. It usually involves coordinating the activities of dozens if not hundreds of developers to a productive end. Each contributor adds something special to the process. Creativity is a critical part of the software development process, and so is discipline.

But, when you’re dealing with an open-source project in which contributions are made by an assortment of developers who have varying skill sets, backgrounds and interests, melding creativity and discipline together in order to make working software can be a difficult undertaking. Hence the beauty of using the pull request executed under an effective branching and release pattern. These practices allow developers to have a great deal of independence, yet provide the safeguards necessary to publish quality software on an ongoing basis.

As we’ve seen in the investigations described above, branching and release patterns tend to be fairly consistent among most of the projects examined. Yet, there are differences. This is not a bad thing. All software projects are different. It’s the nature of the beast.

The trick to having an effective branching and release pattern is to devise one that is consistent and easy to follow. Ease of use promotes broader adoption. An important part of “ease of use” is that the branching and release pattern must be understandable. For some projects, such as Kubernetes, Ansible and Envoy, which have a fairly complex approach, this means written documentation. For a smaller project, such as Deno, the simplicity of its branching and release pattern can be shared in an email or by word of mouth. No matter the method of communication, what all projects have in common is that branching and release are not done in an ad hoc manner. There is a pattern, and it’s followed.

Having a consistent and well-known branch pattern is a key factor when intending to make working software in an effective, sustainable manner. If you’re about to start a project and have yet to establish the branching and release pattern that your team is going to use, such a discussion is worth having before you start writing code, not after.

DEV Community: Travis CI

Travis CI Pipelines: 2 Approaches to Source Control Branching

Branching from repo

Branching from fork

Advantages and tradeoffs

Branching from repo is well suited to private governance

Branching from fork is well suited to open-source development

Putting it all together

Configuring Travis CI to Run a Deno Project

Getting started

Travis CI 101

Getting Travis CI to support Deno

Join the initiative

The Cookbook: Build Matrix

Build Matrix and setting up the .travis.yml

Env Vars

Dependency installation using Composer

The Build Matrix

Running the scripts/tests

Other services

DB Init

.travis.yml in it's final form

Activate webhook

Congratulations

The Cookbook: Jekyll

Getting started with Jekyll

Using Bash

Travis and adding Travis to your project

Encrypted credentials

Adding your encrypted vars

.travis.yml

Conclusion

The Build, Test, Nuke Pattern

The Build, Test, Nuke Pattern

Understanding Build, Test, Nuke

Putting Build, Nuke, Test into Action

What is Deno?

Putting It All Together

The Cookbook: Fable

Build Fable with Travis CI

The package.json

The Cookbook: Branch Flow

Branch Flow

Deploying and Using WebAssembly Under Deno on the Server Side Using Travis CI

Getting started

Understanding the demonstration application

Examining the WebAssembly code

Running WebAssembly under Deno

Working with the .travis.yml file

Putting it all together

The Cookbook with Bash

Getting started with Bash (gh-pages example)

Implementing this in your .travis.yml

The Cookbook: Getting started with R

Getting started with R

Implementing Travis CI

Unit Tests

Finishing up

Anchore Policy Enforcement with Travis CI

Travis CI Pipelines: Anchore

Implementing Anchore into your project via the .travis.yml file to check your containers

Further setup

Anchore Policy Enforcement

Why use security scanners for containers?

A Short Journey into Source Control Branching and Release Patterns

A Short Journey into Git Branching and Release Patterns

Kubernetes

Ansible

Envoy

Deno

Putting it all together

The `package.json`

Working with the `.travis.yml` file

Implementing this in your `.travis.yml`

Implementing Anchore into your project via the `.travis.yml` file to check your containers