DEV Community: Bruno Paz

How to keep software installed from GitHub updated, with Ansible.

Bruno Paz — Fri, 11 Jun 2021 14:22:55 +0000

The simplest way to install software on Linux based Operating Systems is to use the default package manager that comes with the OS, like apt in Debian/Ubuntu or dnf in Fedora.

Submitting software and maintaining these repositories is not trivial for the common developer, and so there are lot´s of great software that is simply not available or is outdated on the package managers repositories.

Flatpaks and Snaps are growing as a viable alternative to distribute software, but still, there are tons of great Software that the only way to get it, is from GitHub.

If the repository has a good release process, installing software from GitHub can be almost as easy as using a package manager.

Many projects provide artifacts in multiple formats like deb, rpm, AppImage and you can choose the most convenient way for your system.

If that is not available, at least you should have a tarball that you can download and extract it´s contents to the appropriate place on your System.

The biggest issue with installing software directly from GitHub is the update process.

When you use any kind of package manager, updating all software is a matter of running a single command like apt update or equivalent.

For software installed from GitHub, the process is a lot more manual as you have to go to the GitHub repository, check if there is a new release, download and install the correct artifact and repeat the process for each software that you have installed this way.

What if we could automate this process? This is where Ansible comes in.

What is Ansible?

Ansible is a tool for automating cloud provisioning, configuration management and application deployment. It´s mostly used by SysAdmins/DevOps to manage multiple servers as infrastructure as code, but the abstractions are the same whether you're managing one machine or a hundred.

Under the hood, Ansible is just a domain specific language (DSL) for a task runner that runs over ssh. You write Ansible yaml files which describe the tasks that you want to run on each machine and Ansible will take care of running these tasks and ensure that the state of your system matches, matches what is specified.

Ansible includes many built-in modules, like "copy" or "command", that abstracts common actions like copy files or execute shell commands. You can check all the available built-in modules here.

There are also many 3rd party modules, built by the community, that you can use.

In this article we will use the GitHub release module, which provides an interface to the GitHub Releases API, and will allow us to get the latest release of a particular repository, in our Ansible tasks.

Ansible is very powerful, and we will only surface the basics on this article. I think it´s important to explain some terminology before going further:

Playbook: A playbook consists of a set of tasks that you want to run on the target machine. It´s the entry-point for the Ansible execution. These tasks can be defined directly in the playbook file or included from other Ansible files or roles.
Role: A role is the primary mechanism for breaking a playbook into multiple files. This simplifies writing complex playbooks, and it makes them easier to reuse. For Example you can have a "docker" role that installs and configures docker, and use that role in your playbook, instead of repeating the same tasks in all your playbooks that needs docker.
Module - A module is a reusable, standalone script that Ansible runs on your behalf, either locally or remotely. Modules interact with your local machine, an API, or a remote system to perform specific tasks.
Inventory - A list or a group of machines/hosts where Ansible will be run against.

Install GitHub Cli using Ansible

To demonstrate the use of Ansbile to install software from GitHub, we will create simple playbook to install the GitHub CLI tool, on an Ubuntu based system. You can find the full source code at GitHub.

Installing Ansible and dependencies

Ansible is available in the default repositories for most popular distros. You can also install it using Python Pip.

For ubuntu 20.04, we could install it with the following commands:

sudo apt-get update
sudo apt install -y python3-pip ansible
ansible --version

You can check the specific instructions for your distro on the Installation Guide in Ansible Docs.

We also need to install the github_release Ansible module. This module is part of the "community.general" collection, that you can install using Ansible Galaxy, a package manager for Ansible content. A Collection is a distribution format for Ansible content that can include playbooks, roles, modules, and plugins.

Ansible Galaxy is included when you install Ansible. You can run the following command to install the respective collection:

ansible-galaxy collection install community.general

This Ansible module depends on the github.py Python package to interact with the GitHub API. It´s a Python package, so we can install it with Python Pip.

pip3 install github3.py

Creating the playbook.

Creating an Ansible playbook is as simple as creating an YAML file following the structure required by Ansible.

The full playbook for this example, will look like this:

- name: GitHub Cli install
  hosts: all

  vars_prompt:
    - name: github_token
      prompt: "What is your GitHub Token?"
      default: "{{ lookup('env','GITHUB_TOKEN') }}"
      private: yes

  tasks:
    - name: "Get Latest Release from Github"
      community.general.github_release:
        user: cli
        repo: cli
        action: latest_release
        token: "{{ github_token }}"
      register: release

    - name: Print Latest release
      ansible.builtin.debug:
        var: release

    - name: Download Binary
      ansible.builtin.unarchive:
        src: https://github.com/cli/cli/releases/download/{{release.tag}}/gh_{{release.tag[1:]}}_linux_amd64.tar.gz
        dest: /tmp
        remote_src: true

    - name: Install Binary
      ansible.builtin.copy:
        src: /tmp/gh_{{release.tag[1:]}}_linux_amd64/bin/gh
        dest: "/usr/local/bin"
        mode: a+x
      become: true

The hosts property is used to specify in which machine(s) this playbook will be run. This is mostly useful, for multiple servers orchestration, as you might want to run different tasks on different hosts, depending on the role of the server for example (web, database etc).

Since this will be run only on a single machine, we can use the keyword "all".

The var_prompts section, allow us to specify a list of variables that Ansible will prompt the user before running the playbook. In this case we will ask for a GitHub token, defaulting to the value of "GITHUB_TOKEN" environment variable.

This isn´t really needed for this simple example, but if you are doing many requests to GitHub API in the same Ansible run, it might be wise to set it, to avoid rate limits of the GitHub API.

The tasks section is where we specify the commands that we want Ansible to run on each host. These commands will be defined with Ansible modules.

Each task will be executed in the defined order.

So, If we wanted to manually install the GitHub Cli from GitHub, these would be the steps that we would do:

Go the the releases page to see what is the latest available release.
Download the appropriate artifact for our system.
Depending on the format of the artifact, we could install it directly, or if a tarball, for example, we would need to extract it and move the contents to the appropriate place.

Let´s see how we can automate these steps with Ansible.

To get the latest release, we will use the "github_release" module we installed before.

  - name: "Get Latest Release from Github"
    community.general.github_release:
      user: cli
      repo: cli
      action: latest_release
      token: "{{ github_token }}"
    register: release

We specify some options like the "user" and "repo" and also the "register" property is used, so we store the output of the command in a variable to be used in the next steps.

We can see the value of the variable for debug purposes, using the "debug" module.

- name: Print Latest release
  ansible.builtin.debug:
    var: release

Then we need to download the respective artifact from GitHub. This step can be a little different depending on the project and how they create the releases.

As we can see in the releases page, this project offers artifacts for deb, rpm, and archive (tar.gz). For this example, we will use the archive version as it is the most common used format that we will probably find. But if the project have a better format available for your system, you should use that. If you are using Ubuntu, you could use the APT module to install the .deb file directly instead.

For downloading and extracting the tarball, we will use the built-in Unarchive module, that can do both at the same time:

 - name: Download Binary
   ansible.builtin.unarchive:
    src: https://github.com/cli/cli/releases/download/{{release.tag}}/gh_{{release.tag[1:]}}_linux_amd64.tar.gz
    dest: /tmp
    remote_src: true

Here we use the release variable that we saved in the previous step, to construct the full download link for the archive. The unarchive module will automatically download and extract the file specified in the "src" property to the directory specified in the "dest" property. The "remote_src" flag is needed to indicate Ansible that the source is a remote URL.

After we download and extract the artifact, we can move the respective executable file to a place in your PATH, like /usr/local/bin.

We can use another built-in Ansible module, "copy" do to that:

    - name: Install Binary
      ansible.builtin.copy:
        src: /tmp/gh_{{release.tag[1:]}}_linux_amd64/bin/gh
        dest: "/usr/local/bin"
        mode: a+x
      become: true

The file needs to be executable, so we specify the "mode" property to indicate the respective permissions. The "become" property is used to run the command as sudo, since the "/usr/local/bin" is usually owned by the root user.

And that´s it.

To run the playbook, open a terminal in the directory where your playbook and hosts file is located, and run:

ansible-playbook -i hosts setup.yml

The -i flag indicates a path to the "inventory" file, which specifies the ip addresses and other connection properties that will be used by Ansible to connect to the target machine. Ansible works by connecting to the target machine via SSH, but since we are running this playbook locally, we can use the property "ansible_connection" to indicate that in the hosts file.

local ansible_connection=local

After Ansible is executed , when we open a new terminal and type gh, it should show the GitHub CLI help command.

Using it at scale.

This example, showed the basics of using Ansible to install GitHub software. The tasks will vary slightly depending on the project and what kind of artifacts they provide.

While the example installs a single piece of software, you could do exactly the same logic to install many software from multiple GitHub repositories in the same Ansible playbook.

You could define all your tasks in the playbook file, but like in programming, when your playbook gets too big, it´s recommended to extract into seperate files. Think of the playbook file as the "main" function of your program. You can do this by using roles and create a single role for each software, or you can just create seperate yaml files, each one containing the tasks for each software and then using the Include tasks directive, to include the tasks in the main playbook file.

Roles are most useful if you want to reuse functionality across different playbooks or share with the community and also if your tasks are more complex or requires same more configuration.

In our example, we could encapsulate all the tasks defined in the main playbook into a role and then reference it from the playbook file like this:

- hosts: all
  roles:
     - { role: github-cli }

See this example of a role that installs the AWS Cli.

I am using "include_tasks" in my setup and use normal folders to organize each software as most of the tasks are very simple, but if I was starting today, I would probably use roles, which is a more "standard" way and could allow me to share them to other persons.

Conclusion

GitHub is a great source of software, but keeping all of it updated with the latest release, can be a very manual process and a lot less convenient that using your distribution package manager. Ansible can help automating that bit.

Create a playbook and tasks for each of your software, and then every time you want to update your system, simply execute the Playbook, in a similar way as you would run apt update or flatpak update. The "github_release" module will take care, of getting the latest release directly from GitHub, so every time you run the playbook, you will install the most recent version of the Software.

Ansible can be used for a lot more than just installing GitHub software. You can completely automate the setup of a new machine with it. Check my personal setup here for inspiration.

Thanks for reading.

Serverless beyond FaaS

Bruno Paz — Tue, 07 Jul 2020 22:01:30 +0000

Microservices are dead!! Serverless is for some time, the new hype in the infrastructure world.

One of the main advantages of a Serverless Architecture is that it frees developers from having to worry about server maintenance, scalability, etc, as these will be handled automatically by the Serverless platform provider, so they can focus entirely on building great products.

This is great, mostly for single devs and small teams as server management, even at a small scale, can become a complex and time-consuming task.

One particular type of Serverless, and perhaps the most popular one, to the point it´s often mixed up with the concept itself, is Functions as Service (FaaS), also known as lambdas, due to AWS Lambda, which is one of the most popular providers of this technology.

The main idea behind lambdas is that you build small independent functions, each one responsible for a small part of your application.

Considering a traditional REST API, you can imagine that any Controller Action would be a separate function.

So now, instead of having 50 Microservices, you have 1000 functions! ;)

It´s true that with a FaaS platform, you won´t need to worry about managing the servers, scalability, etc that would take a lot of effort to do it right, in a big Microservices Architecture.

But you still have to deal with all the coordination between the functions.
You will also have extra complexity for devs to develop and test locally.

And because each provider implements its FaaS platform differently, the lock-in is very high.

Besides, the application will have to be designed in a specific way, tied, to a specific deployment style, so you really can´t move back from a full FaaS architecture to another kind of architecture that easily.

The idea of this article is not against Lambdas. There are lots of great use cases for them, like Event-driven applications, very specific tasks like Image Processing, Sending emails, and even as a small backend for static sites.

What I would like you to remember from this article, is that Lambda´s != Serverless and there is Serverless beyond Lambdas!

I have seen devs, trying to squeeze existing applications into Lambdas, like it´s the only way to deploy applications.

Or people creating issues on Open Source projects like Strapi, asking to make it run on Lambda. Or discussing ways to run Laravel on Lambda, or Rails on Lambda.

I understand the appeal. no server maintenance, almost zero cost for small/medium size projects. But we don´t need to be constrained by FaaS limitations to enjoy the benefits of Serverless. There are alternatives.

One great example is Google Cloud Run.

Google cloud Run fits perfectly my vision of Serverless.

You just need to have a Docker image and you can deploy it with a single command:

gcloud run deploy --image "myimage" --platform managed

No need to change the way you build applications. No limitations on technology. If it can be containerized you can deploy it on Cloud Run.

With Cloud Run, you get all the main selling points of serverless:

No need to worry about server maintenance.
Automatic Autoscaling (including scaling to zero, when the application is not in use)
Pay only for what you use.

Without the limitations of Lambdas.

I think we need more platforms like Cloud Run, with I feel like an evolution of more traditional PaaS like Heroku.

Cloud Run, is build on top of Knative. which is an open-source Kubernetes-based platform to deploy and manage modern serverless workloads, so others could use the same foundations.

I know AWS has Fargate, which seems to require some more configuration but should be the closest equivalent of Cloud Run in AWS.

I am also looking forward to what comes out from the DigitalOcean/Nanobox integration.

Conclusion

Serverless is a great concept and I believe will keep maturing in the next years.

It won´t replace traditional servers for more complex applications or when more control is needed.

But, for smaller teams and single developers, who want to build a product without having concerns about scalability and server maintenance and with the extra benefit of only Pay for what is used, is perfect.

It´s important to understand that Serverless is not just FaaS. Platforms like Google Cloud Run are a great example and might be a much simpler alternative for deploying your applications instead of trying to squeeze them into a Lambda.

Paid Developer tools you can´t live without?

Bruno Paz — Fri, 03 Jul 2020 15:09:11 +0000

In general, I believe most Developers always tend to look for using Free or Open Source Tools.

And the fact is, thanks to the amazing community of Open Source Developers, you can have all the essential development tools for free!

A personal fact, and maybe it´s something that comes with the age or experience, and by getting a better understanding of how valuable development time is, I am realizing that I am more open than ever to pay a fair price for top quality software, sponsoring other Devs, etc.

So I decided to write this article, to open the discussion: What paid software do you use on a daily basis that you can´t live without?

It doesn't need to be a Software per se, can be some site template you bought, for example. Some resource or tool that you gladly paid for it, that makes your life as Dev much easier.

I can start naming a few:

Jetbrains IDEs

While there are awesome Open Source Code Editors like VS Code, JetBrains IDEs are very popular and powerful.

Some of the features I highlight are the powerful refactoring tools and much more reliable and faster code completion.

I use a mix of VS Code and Jetbrains IDEs (Goland, PHPStorm, WebStorm) but for more "serious" and complex projects, I tend to prefer Jetbrains IDEs.

VSCode is closing the gap and has many more contributions in terms of Extensions and I would love to use it fully, but there are some tasks that Jetbrains is still far superior.

It´s definitely a very powerful software that you might be willing to pay for.

Cacher

Cacher is a Code Snippets management tool that helps to organize your snippets library with labels and a powerful search engine and everything is Synced to GitHub Gist.

It´s cross-platform and also provides deep integration with the most popular Editors like VS Code, Jetbrains, Atom, and Sublime.

I couldn´t find anything free with this level of quality and features.

With the usage I am giving to, the personal plan of 6$/month is completely worth it.

Tailwind UI

TailwindUI is not an application but I think it deserves a mention here.

It is a blessing for backend developers like me who suck at design.

It provides a high-quality set of all kinds of components for building any kind of Web UI, from an application to marketing and landing pages.
The components are pure HTML with Tailwind CSS.

You can see what is included here

The price range from $149 to 249$ one-time payment.

It can look expensive but believe me, it´s an amazing toolkit for building good looking websites and you can see the amount of work put on it.

If you want to build a web application but suck at design, it´s a great investment.

Now, I want to hear from you!

And also start streaming your micro-payments. Let´s see how this Web Monetization thingy really works. 😛

How to create your own auto-completion for JSON and YAML files on VS Code with the help of JSON Schema

Bruno Paz — Mon, 27 Apr 2020 20:51:52 +0000

Visual Studio Code has the ability to display autocomplete suggestions for popular configuration files in JSON and YAML format out of the box.

For example, If you have a package.json file opened in VS Code, you can tap CTRL + Space and a pop-up will appear, displaying suggestions for all the available fields for that file type.

This is very practical and provides a quick way of knowing all the available options without having to look at Documentation.

To be able to do this, VS Code uses JSON Schema under the hood.

Note: other editors like the ones from the Jetbrains family, also use the same strategy to provide the same functionality, so, while in this article VS Code is used as an example, the concepts should be applicable to any of these IDEs. Be sure to check their respective documentation.

What is JSON Schema?

JSON Schema is a specification that allows you to describe the structure of a JSON document and validate documents against that schema.

For example, the following schema could be used to describe a "Person" entity:



{
  "$id": "https://example.com/person.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The person's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    }
  }
}

As you can see, a JSON Schema is simply a JSON document with a specific set of tags that are used to describe the document structure.

This is a very basic example. You can check the JSON Schema website for a complete view of the specification.

While JSON Schema is designed to work with JSON files, it can be used to with any other file that can be converted to JSON, like YAML.

VS Code uses the JSON Schema, to infer the document structure of a particular file type and to display the appropriate suggestions as well as validations.

To provide the autocomplete capability to a big range of popular file types, VS Code leverages the JSON Schema Store project, which hosts JSON Schema specifications for more than 200 file types.

The JSON Schema Store provides a very good set of schemas. Still, with so many different file types, of course, it can´t have schemas for everything.

The good news is that you can easily create your own and make it available for your editor to use.

And while you are at it, JSON Schema Store is Open source, so why not contribute to their repository, so that everyone can benefit from it?

To demonstrate this, we will create a schema for Hadolint, a popular Dockerfile linter.

Hadolint supports configuration with a .hadolint.yaml file.

It only allows two configurations to be defined:

ignored - which defines a list of lint rules to ignore
trustedRegistries - which as the name indicates, it´s a list of valid Docker Registries to allow.

Example file:



ignored:
  - DL3008

trustedRegistries:
  - docker.io

It´s a really simple structure, so a good example to start.

Creating our schema

So we need to create the JSON Schema specification for this file type.

First, we need to understand the structure, and to find out all the possible options for that particular file type. This task can be easier or not, depending on the available documentation.

Probably the best way to start is to look at the project documentation or for example files. Some projects provide example files with all the possible values and you can infer the schema from there.

In the case of Hadolint we can get everything needed from the project Readme.

Next, we will create an hadolint.json file with the schema specification.

The shcema will be very similar to this:



{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "JSON Schema for Hadolint, a Dockerfile linter tool",
  "description": "Dockerfile linter, validate inline bash, written in Haskell",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "ignored": {
      "type": "array",
      "description": "A list of rules to be ignored",
      "items": {
        "type": "string",
        "oneOf": [
          {
            "const": "DL3000",
            "description": "Use absolute WORKDIR.",
          },
          {
            "const": "DL3001",
            "description": "For some bash commands it makes no sense running them in a Docker container like ssh, vim, shutdown, service, ps, free, top, kill, mount, ifconfig.",
          },
          {
            "const": "DL3002",
            "description": "Last user should not be root."
          },
          {
            "const": "DL3003",
            "description": "Use WORKDIR to switch to a directory.",
          },
          {
            "const": "DL3004",
            "description": "Do not use sudo as it leads to unpredictable behavior. Use a tool like gosu to enforce root.",
          },
          {
            "const": "DL3005",
            "description": "Do not use apt-get upgrade or dist-upgrade.",
          }
        ],
        "title": "Rule",
      }
    },
    "trustedRegistries": {
      "type": "array",
      "description": "A list of trusted registries. Ex: docker.io",
      "items": {
        "type": "string"
      }
    }
  }
}

Note that I have omitted some options on the "ignored" field for readability.

Let´s break down our schema:



 "$schema": "http://json-schema.org/draft-07/schema#"

This first line describes which version of the schema we are using. We use the "Draft-07", which is the latest version at the time of this article.

We then define a "title" and a "description" for the schema.



 "title": "JSON Schema for Hadolint, a Dockerfile linter tool",
 "description": "Dockerfile linter, validate inline bash, written in Haskell",

The root "type" property will have the value "object" most of the time, and we also define "additionalProperties" property with value "false", to indicate that the file can´t have extra properties that are not listed in the schema.

The "properties" field is where we start defining all the properties of our object.

As we saw, hadolint.yaml has two properties: "ignored" and "trustedRegistries" which are both a list of strings.

For the "trustedRegistries" it´s an open list, while for the "ignored" each option must be one of the rules available in Hadolint.

"trustedRegistries", can be defined as this:



"trustedRegistries": {
   "type": "array",
   "description": "A list of trusted registries. Ex: docker.io",
   "items": {
      "type": "string"
   }
}

We define the "type" as an "array" whose items are of type "string".

For the "ignored" property, it´s similar but has a fixed list of options.

It could be defined as this:



"ignored": {
   "type": "array",
   "description": "A list of rules to be ignored",
   "items": {
      "type": "string",
      "oneOf": [
          {
            "const": "DL3000",
            "description": "Use absolute WORKDIR.",
          },
          {
            "const": "DL3001",
            "description": "For some bash commands it makes no sense running them in a Docker container like ssh, vim, shutdown, service, ps, free, top, kill, mount, ifconfig.",
          }]
     }
}

JSON Schema also has "enum" type. We could have used it instead of type "string" together with "oneOf" field, but as we want to add a description to each value, Enum type doesn't support that.

With this, we have the basic structure of our schema ready.

This example just shows the surface of what you can do with JSON Schema. You can see more complex schemas in Schema Store. for example gitlab-ci.

Depending on the complexity of the file whose schema you are creating, it can be a little boring to specify every available option, but if you work with that file a lot, It can save you valuable time in the future.

Now that we have created the schema, let´s see how we can make it available on VS Code.

Enabling the Schema on VS Code

Since the new schema is not in the Schema Store, you must register it on VS Code. This can be done on the VS Code Settings page.

As we are creating a Schema for a YAML file, make sure you have the YAML Extension installed before continuing.

Then, go to File -> Preferences -> Settings (or use the Command Pallete) to open the settings page and search for yaml.

Open the settings for the YAML extension and search for "Yaml: Schemas" and click "Edit in settings.json".

The "settings.json" file will open. you need to search again for the "yaml.schemas" object. If it doesn´t exist yet, you will have to create it.

This property represents a key-value, where the key is the absolute path to the schema file on our system and the value is a glob expression that specifies the files that the schema will be applied. In my case it looks like this:



"yaml.schemas": {
  "/home/bruno/Code/Personal/other/schemastore/src/schemas/json/hadolint.json": ".hadolint.yaml",
},

Save the file and reload VS Code to finish the process.

If everything worked as expected, when we create a new .hadolint.yaml file and press CTRL + Space, VS Code should then display the suggestions based on the schema we created for this file type.

Note that, it could take some seconds for VS Code to index the schema in the first time.

And that´s it. We have just built an autocomplete feature for hadolint.yaml files.

Conclusion

JSON Schema provides a simple way to implement auto-completion for JSON and YAML files and is supported out of the box, by many popular editors like Visual Studio, Visual Studio Code, and the Jetbrains family IDEs.

JSON Schema Store hosts schemas for many popular file formats, but you can also create your own as demonstrated in this article.

It´s a matter of creating a schema describing your file following the JSON schema specification, and import it in your editor.

The process of creating a schema for a complex file can be a little painful and boring but in the end, I think the gains in productivity if you work a lot with that file type can pay off.

I will finish the Hadolint schema and do a PR for the Schema Store, I hope this week.

Thanks for reading and if you have any questions, feel free to use the Comment Section.

Also, if you like my content, you can Buy me a coffee ;)

Load testing your applications with Artillery

Bruno Paz — Sat, 11 Apr 2020 10:13:47 +0000

In this article, I will show how to use Artillery to load test your application.

What is Artillery?

Artillery is a modern, powerful & easy-to-use solution for load testing and functional testing.

It´s built with NodeJS and it´s open source.

Unlike other load testing tools that have complicated GUIs, Artillery is a simple CLI tool, making it very easy to use and to integrate into any CI environment.

It supports testing applications that use HTTP, Socket.io or Websockets.

One of the biggest points of Artillery is that it allows defining test scenarios using YAML, making it possible to version the tests together with the application code.

Getting started

Artillery can be installed like any other Node package, using NPM:

npm install -g artillery

Doing a quick test

Artillery can be used to do one-off tests to a specific URL in a similar way of tools like Apache Benchmark.

That is done using the quick command.

artillery quick -c 10 -n 20 https://httpbin.org/get

This will launch 10 virtual users that will do 20 requests each, to the specified URL.

The quick command supports some other flags to control how the load will be distributed, like rate, which can be used to define the number of new users per second, or the duration, which defines a fixed time for the test.

For example, the following command will run the test during 10s, with 2 new users arriving each second, resulting in around 20 requests in the total.

artillery quick https://httpbin.org/get -r 2 -d 10

You can combine all these flags, to achieve the desired pattern of load for your test.

In terms of other options, I found this command to be pretty limited as you can´t configure almost anything more of the request, like defining request headers. It also only allows GET requests.

So, if you just want to do some one-off test directly from the command line and need a little more flexibility configuring the request, there are probably better tools like Apache Benchmark.

But the power of Artillery lies in being able to simulate realistic user behavior with scenarios and flows, using declarative YAML files.

Scnearios and YAML configuration

The most basic test case can be defined like this:

config:
  target: 'https://httpbin.org/'
  phases:
    - duration: 60
      arrivalRate: 20
scenarios:
  - flow:
    - get:
        url: "/get"

The config section allows defining some generic configuration about the test, like the target URL, the duration of the test and the way users will generate load.

In this example, we define one phase, which will last 60 seconds with 20 new virtual users (arriving every second (on average)).

The scenarios section defines the scenarios this test will cover. Each scenario has a flow associated with it, which is a set of steps that each user will do in your application.

We will see this in more detail, later in this article.

In this particular case, we have defined one single scenario, with a single step, which is a GET request to the https://httpbin.org/get endpoint.

You can run this test using the run command:

artillery run <path_to_file>.yml

This is very similar to the "quick" example, just a request to a specific endpoint.
But unlike the "quick" command, using a configuration allows you to define a lot more params of the request, like HTTP method, headers, cookies, request payload, etc. You can even load your payload dynamically from CSV files.

Here is a more advanced example demonstrating how to do a POST request with payload data loaded from a CSV file.

  config:
    payload:
      # path is relative to the location of the test script
      path: "users.csv"
      fields:
        - "username"
        - "password"
  scenarios:
    - flow:
        - post:
            url: "/auth"
            json:
              username: "{{ username }}"
              password: "{{ password }}"

This test will do a "POST" request to the "/auth" endpoint, with a JSON body containing the fields "username" and "password", which values are loaded from a CSV file "users.csv", from the specified fields.

You can find out about all the possibilities in the Documentation.

A more complete user flow

We have seen how to define a scenario with a simple flow.
Next, we will see how you can define an entire user journey.

Considering a typical E-commerce site. You might want to simulate a user flow, starting from the catalog page, selecting a product, and add it to the cart.

This can be achieved with the following configuration:

config:
  target: "https://staging1.local"
  phases:
    - duration: 60
      arrivalRate: 5
  payload:
    path: "keywords.csv"
    fields:
      - "keywords"
scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"
      - get:
          url: "/details/{{ id }}"
      - post:
          url: "/cart"
          json:
            productId: "{{ id }}"

Some things we haven´t seen before.

A flow can consist of multiple steps, representing the user journey in the application. Each user spawned by Artillery will execute all the defined steps sequentially.

In this particular example, The first step is doing a POST request to the "/search" endpoint with a body, containing keywords, loaded from a CSV file.

- post:
          url: "/search"
          body: "kw={{ keywords }}"
          capture:
            json: "$.results[0].id"
            as: "id"

The "capture" block allows us to get a value from the response and save it as a variable for later use. You can use "JSONPath", "XPath", "Regex" and more to parse the response and get the value you want.

For example, to get a URL form an anchor tag in HTML response you could do as following:

- get:
    url: "/some/page"
    capture:
      - selector: "a[class^=productLink]"
        attr: "href"
        as: "productUrl"

Back to our E-commerce flow, We will use the captured "id" variable from the "search" step, to go to the "product details" page and to add the product to cart:

- get:
     url: "/details/{{ id }}"
- post:
    url: "/cart"
    json:
       productId: "{{ id }}"

As you can see with a few lines of YAML we can define an entire user flow to test.

Artillery supports many more functions to work with HTTP requests and responses. You can find then in the http reference section of the documentation.

Multiple Scenarios

In the previous example, we saw how to define a single scenario with multiple steps.
But you can also define multiple scenarios to simulate users doing different things at your application at the same time.

Let´s say you want to simulate users searching and buying products at the same time. You can create two different scenarios to test both flows. Note that each user will only execute one of the scenarios.

The scenario supports a "weight" property which you can use to indicate how likely it is that scenario to be picked by a particular user.

By default, each scenario will have the same probability. If you define a scenario with weight 2, that scenario is twice as likely to be picked as the one with 1 (which is the default value).

Setting success conditions

It´s possible to configure Artillery to return a non-zero exit code if the test run doesn't comply with specified conditions based on a set of parameters like error rate, minimum, maximum and percentile based latency.

This is really useful in a CI environment as you can make the test fail if it doesn´t meet your performance requirements.

The following configuration will ensure that at least 95% of requests are executed below 200ms, otherwise, the command will exit with an error.

config:
  ensure:
    p95: 200

Extend Artillery

Hooks

Artillery has support for "hooks", which allow for custom JS functions to be executed at certain points during the execution of a scenario.

The following extension points are available:

beforeScenario and afterScenario - called before/after a virtual user executes a scenario
beforeRequest - called before a request is sent. request parameters can be customized here.
afterResponse - called after a response has been received. The response can be inspected and custom variables can be set here.
function - which can be inserted as a step at any point in a scenario.

You can use these hooks, to log extra information you need or to modify the request of some sort before doing a request.

this article shows a very good example of this, by leveraging Faker.js to generate random data for the tests.

Plugins

Artillery supports custom plugins that you can write using Javascript.

Artillery has some official plugins like artillery-plugin-expect that can be used to add expectations/assertions to your HTTP scenarios, allowing you to do some basic functional tests together with your load tests or the artillery-publish-metrics, which allows sending metrics about your test runs to external services like Datadog, InfluxDB or StatsD.

Here is an example of the artillery-plugin-expect plugin in use. See The "expect" section:

scenarios:
  - name: Get pets
    flow:
      - get:
          url: "/pets"
          capture:
            - json: "$.name"
              as: name
          expect:
            - statusCode: 200
            - contentType: json
            - hasProperty: results
            - equals:
              - "Tiki"
              - "{{ name }}"

You can build your own to add extra functionality.

Go distributed with Artillery Pro.

Besides the Open source version, Artillery has a Pro version starting at $199/month.

The pro version offers deep integration with AWS services and allows us to run distributed load tests using AWS infrastructure, and includes other integrations designed to support testing of modern enterprise applications & modern DevOps workflows.

The open-source version should be enough for most people, but if your scale is pretty big, and you reached the limit of load that a single machine can generate, these extra features can be valuable.

You can find more about the Pro plan here.

Conclusion

There are many load testing tools out there, some a lot more popular like Jmeter or Gatling.

Where Artillery really shines for me compared with these tools, is the balance between simplicity and ease of use and functionality.

Being a simple CLI tool makes it very easy to install and use in every environment, from the local developer machine to the CI server.

It´s built with Node, which makes it a lot lighter these Java-based tools.

The tests are declarative in YAML files, so they are easily readable by everyone and can be versioned together with the application code.

Scenarios and flows provide great flexibility and allow you to go beyond testing a single endpoint at a time and test entire user journeys.

"Success Conditions" helps to keep your performance requirements under control, by integrating with your CI tool and make the build fail if it doesn't meet your performance criteria.

Hooks and plugins make possible to extend the tool with extra functionality.

There are no more excuses not to write load tests for your applications!

My 2019 Personal Tech stack for Web Development

Bruno Paz — Sun, 01 Dec 2019 12:18:27 +0000

In this article, I will talk a bit about my personal tech stack and related tools for Web Development.

But, before starting, let me just do a brief introduction about myself.

Who am I?

I am Web Engineer, working atm in an E-Commerce/Payment company where I do mostly Backend development, using Golang and Java as core technologies.

I started programming in 2011 after graduating in Computer Science.

I was still looking for my real passion, and working in a very small startup as my first job, forced me to do a little bit of everything from frontend to backend and some basic ops, in the old days where HTML, CSS, and jQuery were enough to build a frontend.

Over the years I specialized more in Backend development, mostly using PHP and Symfony.

Backend development, software architecture and engineering practices are my main focus and passion, but I want to understand all the areas of the Software Development Life cycle and be able to build and launch any product on my own, "from idea to production".

That´s what I aim for, and try to balance my learnings to reflect that.

It´s almost impossible to be a complete expert full-stack dev these days. The tech world is just too broad. but I also don't believe in "Single Language" developers.

The ability to learn and adapt fast is an essential skill.

I think the sweet spot is somewhere in between. I identify a lot with the concept of T-Shaped developer.

Nowadays, the term full-stack developer is also very subjective as there are many more areas besides the traditional Frontend/Backend.

I like this definition by Adam Wathan:

// Detect dark theme var iframe = document.getElementById('tweet-1186388252745457665-42'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1186388252745457665&theme=dark" }

And this is the kind of full-stack I want to be.

In the next sections, I will present some of my favorite technologies and tools for all the areas of Web Development.

Bear in mind, that I am not an expert on all of these. Far from it.

The backend technologies and some infrastructure yes, I have deep experience and use them a daily basis.

Others, it´s more a theoretical knowledge from reading articles and books and small experiments, but I am convinced of the value of these and plan to use them in real projects when I have a chance.

Backend

So let's start with my main area of expertise, Backend development.

I have two main languages in my stack. PHP and Golang.

PHP and Symfony

PHP and Symfony are the technologies that I am more comfortable with, since I worked with them for the most part of my career, from the old PHP 5.3 and Symfony 1.4 to the shiny PHP 7.3 and Symfony 4.

PHP might not be the most popular and pretty language today, far from it, but it´s a very mature language that has received some great improvements in the past few years. It also has some great libraries and frameworks and a strong community.

It´s a "boring" language and IMO, having a boring language in your stack, that you can rely upon to build working software fast without thinking too much is a must-have for any developer.

I wouldn't probably be so keen on PHP without Symfony tough.

Symfony is an amazing and mature framework that focuses on good OOP design and Design patterns. It has very good documentation and a strong focus on Developer experience and performance.

If you like OOP, Design patterns and Clean code, Symfony is a joy to work with.

I am not working with Symfony anymore at work, but I have worked for many years and I can build products very fast with it.

Golang

I have been working with Golang professionally for more than one year and it´s been a fantastic experience.

I think it complements PHP really well, as some of the strong points of Go are precisely some of the PHP weakest points like concurrency and long-running processes.

Go it's incredibly light-weight and strongly typed, which I like. It´s not really an OOP language but has some powerful concepts like interfaces that help to build well structured and modular codebases.

Goroutines are awesome for building highly concurrent applications.

Being a compiled language makes it easy to distribute and install, making it one of the best languages ideal for Command-line applications, System Tools, and utilities and microservices and APIs.

Go took the world of DevOps tooling by storm, being the language of choice for popular software like Kubernetes, Docker or Terraform

I am writing more and more Go and it´s becoming my primary language.

With Symfony and Go in my stack, I believe I can solve 99% of problems related to Backend development.

The future

I will continue using more and more Go, but I will always have Symfony in a special place in my heart.

Typescript is a language that I have an eye on as it gets more and more popular.

I have never really done everything with it, but if one day I want to replace PHP as the OOP language in my stack, Typescript is looking like a strong candidate.

I would also like to play with a more functional language. Scala, Haskell, not sure yet.

Frontend

One of my goals for next year is to improve my front-end skills.

With a new JS framework every week, and me not wanting to become a frontend developer, choosing a stack that is stable and mature, with good backward compatibility records, good documentation, and a big community around it, are very important factors to look into.

Right now there are two major ecosystems, who met these criteria: React and Vue.

I choose Vue because I prefer the way it is designed with templates instead of JSX for example and I think it´s easier to understand.

Vue has a great ecosystem of tools and libraries.

With Vue itself for Single page applications, NuxtJS when there is a need for Server-Side rendering and Gridsome for more static sites, I can build all kinds of frontends.

I could even hook Nativescript to build native mobile applications.

If you prefer the React ecosystem, the same kind of tools exist. Next.js instead of Nuxt and GatsbyJS instead of Gridsome.

Some of the React tools are more mature, like Gatsby compared with Gridsome, but still, I hoping them to catch up.

A big pain for many backend developers is CSS. I also really suck at design.

Tailwind CSS is a great option and I am looking forward to the components of Tailwind UI.

Or you could go with classic Bootstrap and buy some themes from sites like Themeforest.

Scripting

Every developer needs to do some kind of utility scripts once in a while, from data migration utilities to deployment scripts.

Python is one of the most popular languages for these kinds of scripts because it´s really simple to learn and comes pre-installed in many systems.

But Go also fits really nice on this. You can run it locally with a simple "go run" and you can compile it in a single binary with all the bundled dependencies (which is a pain in python), to run on a remote system.

I can write Python, but since I have much more experience with Go, it´s becoming a natural choice for this kind of scripts.

With a little help of bash for more basic things ;)

Datastores

Almost every application needs to store data in some form.

I use MySQL for Relational data, Redis for in-memory database, cache, etc and ElasticSearch or Algolia for full text search.

Cloud Firestore is really interesting when I don't want to worry about setup a database and NoSQL storage is enough. It´s really great for prototypes, small personal apps, and frontend/mobile-focused applications.

I am starting to look more int some "Cloud-native" solutions like FaunaDB, ArangoDB and CockroachDB.

Messaging

There are many use cases where a Message Queue system is useful like long-running processes or communication between different services.

At my current job, I use Apache Kafka and it works great for complex, event-driven systems.

For personal projects, in most cases, I want a simpler and managed solution. This is where Amazon SQS and Google Cloud Cloud Pub/Sub comes in. They are free for most basic workloads.

Object storage

For storing user uploaded files, Google Cloud Storage and DigitalOcean spaces are two good options.

Another great service, more specialized in media upload is Cloudinary.

CMS and BaaS

To provide content editing capabilities without much effort, Netlify CMS is great for a Git-based workflow and Prismic for an API like workflow.

For a self-hosted solution, Directus or Strapi.

Infrastructure

Code that is not in production is not really useful.

I believe every developer should understand the basic concepts of infrastructure and application deployment.

I am a big fan of the concepts of "Operate what you build" and "Full-Cycle developers" of Netlfix, where the team that develops a system is also responsible for operating and supporting that system.

You can read more about it in this blog post.

Cloud services make all this easier for developers by abstracting the low-level complexities of networking, storage, etc.

For generic all-purpose hosting, DigitalOcean is my platform of choice. It´s very developer-friendly and provides great services at affordable prices.

Their services offering is not as rich as bigger players like AWS or Google Cloud, but the introduction of managed Kubernetes clusters and managed databases were a great step forward.

I am also pretty curious about their future PaaS after they have acquired Nanobox earlier this year.

In the cases where DigitalOcean doesn't cover all my needs, I go to Google cloud.

Tools like Cloud Run, Cloud Functions, Container Registry, Cloud Pub/sub, Cloud scheduler and Cloud firestore provides the foundations for building all kinds of applications.

These tools have a great free tier, allowing building small projects for free.

For provision infrastructure Terraform and Ansible work very well.

When I just need to deploy a frontend or a static site, I use Netlify.

Namecheap to manage my domains and Cloudflare for DNS and CDN.

For monitoring, both Google Cloud and DigitalOcean offer good basic monitoring built-in.

As an alternative Datadog and Sentry are some great tools.

I am still looking for a good Logging Solution. Logz.io looks nice and it´s based on the ELK stack which is very popular and what I use at work. Their Community plan is also very generous for small projects.

UptimeRobot and Opsgenie to guarantee I get notified when a site is down.

Tooling

I am using GitLab for private projects and CI/CD, but with the available free plan for private repositories and the introduction of GitHub Actions and Package registry, I might move everything to GitHub. It´s a tough decision as I love the Product vision of GitLab, but GitHub is still more polished in some areas.

Codacy gives me automated code reviews & code analytics and it´s free for 4 users.

For coding, I use VScode, together with JetBrains IDEs.

Docker and Docker-Compose provides a streamligned development envrionment.

Postman for testing REST APIs, Cacher for storing code snippets, Dbeaver for connecting to databases.

Conclusion

I really like my stack and I believe I can build almost anything web-related with it.

It´s not a static list.

In tech, everything evolves at a very fast pace.

It´s important to keep updated and experiment with new tech on a regular basis and my stack will keep evolving based on that.

But it´s very important to avoid "Hype Driven development".

Always have some "boring" mature tools in your stack, which allow you to keep building things in a fast and sustainable way.

In the end, people are what matters, so focus on building useful things that improve people´s lives, independent of the stack!

Make your project README stand out with animated GIFs/SVGs

Bruno Paz — Sat, 02 Nov 2019 18:26:12 +0000

An image is worth a thousand words and sometimes it´s much easier to demonstrate how something works by using images or videos instead of just text.

A common technique that works very well with GitHub and Markdown, is to use an animated GIF or SVG files.

this or this shows this in action.

Looks cool, doesn't it?

The first thing a user sees in a GitHub project is its README. A good first impression is essential and can make a difference between keep reading or just close the page and look somewhere else.

With an animated image, the user can see immediately how the project works like, without having to read the entire README or to download the entire project to test himself locally.

This is much more engaging experience and personally, If I see a project with this, I am much more likely to keep reading with attention the docs and to use it.

On the other side, for some kind of projects like web applications, it´s almost always a no go, if I cant find quickly any kind of visual demo. It can be just simple screenshots but I want to see how it looks like.

I don't want to clone the repo, set it up, just to see if it´s worth visually.

But how can you create these animations?

Recording your terminal

If it´s enough for you to record your terminal window, you can use asciinema and svg-term-cli for it.

asciinema is a free and open-source solution for recording terminal sessions and sharing them on the web.

To install it, follow the installation guide for your operative system.

Then, open a terminal session and type:

asciinema rec myrecord.cast

This command will start a recording session and every command you type from now on will be recorded. To stop the recording, press CTRL+D.

Your recording will be saved with the name you specified when running the "rec" command, in this example "myrecord.cast".

Asciinema allows you to upload your recording to their website, but you would need to include their player in the project README, which doesn't play automatically and forces the user to click and be redirected to the Asciinema website to see it, so we will use another tool, svg-term-cli to generate a regular SVG file from the Asciinema recording.

svg-term-cli is a node tool, so you need to have node installed on your system and install it with NPM:

npm install -g svg-term-cli

This tool can take an Asciinema recording as input and generate an animated SVG file as output, that you can add to your project README file as a regular image.

To generate an SVG file from our previously recorded session, run:

cat myrecord.cast | svg-term --out myrecord.svg --window

svg-term-cli supports a number of flags that you can use to customize the output image. Please look in the project README for details.

And voilá. You have an animated SVG with your terminal session that you can include in your projects README file as a regular image.

Recording your entire screen

If you need to record more than your terminal window, you can tools like Peek for Linux or recordit for MAC or Windows to Record your screen and export as an animated GIF file.

GIF files can become very big easily, so try to record just the area of the screen you need and by a short amount of time.

You can then add it to your project README as a regular image.

Factors to consider when adding third party dependencies to a project

Bruno Paz — Sat, 02 Nov 2019 16:23:45 +0000

Avoid reinventing the wheel is a good practice when developing any Software project. The focus should be on adding value to the end-user and not to re-implement that function to sort an array every time.

Thanks to open-source, there is a huge ecosystem of available libraries, that can solve most of the common problems you might have.

It can be very tempting for every task, to just look for an existing library that can do it for you and add it as a dependency to your project. Don't reinvent the wheel right? Well, yes, but every extra dependency comes with a cost and it is not something that should be done lightly.

When adding a third-party dependency, your project is now using code that you don't own.

What happens if that code has a bug or a security issue? Will it be fixed fast enough? And updates? Are they backward compatible or will break your application every time? Considering you will get any updates at all. The developer might just stop working on the project. Do you understand exactly what that dependency does and how to use it?

These are all important questions that you have to ask, but even before that, it´s important to understand if you really need it to use an external dependency for the functionality you are trying to implement.

May be is something you can write in a few lines of code? Or maybe you just need one function of a more complex library. In that case, couldn´t you just copy-paste it to your project without having to add the whole thing?

Taking the extreme example of NPM packages like is-odd or is-even. It´s one line of code of basic math!

Node community is known for abusing the usage of third party dependencies. You might have heard about the left-pad incident, where a function of a couple of lines broke half of the internet!

So, think well before adding an external dependency.

If you are decided that you really need to rely on an external dependency, it´s very common to find different implementations that solve the same problem. How to choose between them?

I will enumerate some factors that I consider important in the next sections.

Popularity

If a library is used by many people and has a big community around it, in general, it can be a good indicator of its quality.

A common used metric for popularity is the number of GitHub stars.

But be careful with this. Don't simply consider it as your unique decision factor. A library that was built by a big company or by a well-known developer, can have many more stars (because of more publicity for example), than a one done by a single developer or a smaller org. Or maybe it´s a library that exists at more time.

This criterion alone doesn't guarantee that is a better choice.

It´s a good indicator yes, but should be considered together with the other factors that I will talk about next.

Activity in the Repository

You don't want to depend on something that it´s not actively maintained anymore.

The frequency of commits, number of open issues and PRs and the speed that the PRs are merged are important indicators of project health.

Low activity in the repository can be a signal of worry but it depends on the use case. There are libraries that are so simple, that they don't really need changes every day or month. They just work.

Still, In most cases, I would avoid using a library that have very few commits in the last couple of months and/or with many open Issues / unmerged PRs.

Project maturity and versioning strategy

It´s best to depend on something that is stable and has a well-defined release schedule and versioning strategy.

Projects in the early stages of development might have breaking changes more often and can be riskier to use. The same with projects who don't have a defined versioning strategy like Semantic Versioning.

With SemVer you can have a better understanding of what´s have changed from one version to another and if it´s a bug fix or new feature, so you can plan ahead when to update your library version.

Who is the maintainer/core contributors

Another important factor to consider is who is the maintainer(s) of the project. Is it a known developer or a big organization? or just a single developer working in its free time?

In general, I would say that a project that is behind a big company or with a big number of contributors, can be a safer bet in terms of future maintainability and support, but again, it´s just an indicator, not rocket science.

This fact was indeed often mentioned when favoring React against Vue.js. React is a Facebook project, unlike Vue, where most of the code was done by a single man.

Still, the fact is that Vue has matured a lot and it´s one of the most popular JS frameworks right now, even surpassed React in terms of GitHub stars.

An opposite example is Polymer Project. A framework created by a big company like Google that never gained any traction.

Quality of Documentation

If you want to use a particular library, you have to understand how it works.

A library with poor or no documentation, it´s a big red flag for me.

At very minimum, it should have a README file explaining how to install and with code examples for the main use cases.

Test coverage

If a library has very few or no tests, how can you be sure that it will do what you are expecting it to do and that you won´t encounter critical bugs when integrating it into your project?

It´s really bad to have a bug on your application because of a bug on a third-party library.

Good test coverage can also show that the developer cares about the quality and maintainability of the software and that it's not just some experimental side project and can be used in production.

Number of dependencies

You are adding a dependency to your project, but that dependency will probably have their own dependencies.

This is something that many developers don't pay enough attention to. What are those dependencies? Are those still maintainable and trustworthy? Of course, it will hard to check every dependency but if a library has too many dependencies can be a warning sign.

In case of doubt, I will always choose the one with fewer dependencies.

Conclusion

Avoid reinventing the wheel is essential and thanks to all the amazing Open Source projects out there, our lives as developers are a lot easier as most of the common problems were already solved by someone else.

But adding a third-party dependency to a project is a decision that should be well thought, You project now depends on code you don't own or write. What if it has bugs? Will it be fixed soon enough? Will updates break your code? What if the developer just stops working on the library?

You can mitigate these issues by being rigorous when choosing the dependencies that you will add to your project. Give preference to the ones that are actively developed, have great documentation and test coverage and few extra dependencies.

A good strategy to minimize the dependency on a library in a project is to create a wrapper around it. This will make it easier to change as your application code will never call the library directly, but the wrapper. If you want to replace that library with another, just change the wrapper to use the new library internally and it´s done.

Another important thing is to keep your dependencies up to date. Tools like Dependabot or Renovate can give a big help with that.

What about you? Do you have any criteria for choosing a library that I didn't mention? Feel free to comment below.

Thanks for reading.

Building a basic CI/CD pipeline for a Golang application using GitHub Actions

Bruno Paz — Mon, 19 Aug 2019 20:45:42 +0000

GitHub has announced last week exciting new features for GitHub Actions, including built-in support for CI/CD pipelines.

You can watch the full announcement on Youtube.

This is a huge milestone for GitHub and one of the most anticipated features since platforms like GitLab and Bitbucket already have solutions for this for many time.

It´s another good example of the tremendous evolution of the company under Microsoft and Nat Friedman leadership.

GitHub Actions will allow building a complete CI/CD Pipeline, deeply integrated with the GitHub ecosystem, without the need to use a third-party service like Travis CI or Circle CI, following the trend for "All in one" solutions where GitLab is probably the best example.

The feature is in beta for a limited group of users and is expected to be released for all users in November this year. It will be free for Open source projects and have 2000 free build minutes per month for Private repositories. (by user, not repo).

You can request early access now. I already have it and I am writing this post to show what you can do!

What we will build

To demonstrate the new features of GitHub Actions, we will build a "Hello world" Golang app with a very basic pipeline that after each Pull Request or push to master branch, will lint our code, run unit tests and generate code coverage report using Codecov.

Then when a new "tag" is created in the repository, it will create a new GitHub release using GoReleaser tool.

The project code

The example repository is accessible here, Feel free to fork it or just follow along.

I won't go into many details about the code of the application itself. It´s a standard "Hello world" app that prints the text "Hello GitHub actions" to the standard output.

Here is the code for the main.go file:

package main

import (
    "fmt"
    "github.com/brpaz/go-github-actions/hello"
)

func main() {
    fmt.Println(hello.Greet())
}

And here is the "Greet" function:

package hello

// Greet Greets GitHub Actions
func Greet() string {
    return "Hello GitHub Actions"
}

And the respective unit test:

package hello

import "testing"

func TestGreetsGitHub(t *testing.T) {
    result := Greet()
    if result != "Hello GitHub Actions" {
        t.Errorf("Greet() = %s; want Hello GitHub actions", result)
    }
}

The pipeline

GitHub Actions are based on the concept of Workflows. A workflow is nothing more than a set of jobs and steps that are executed when some condition or event is met. (Ex: a push to the repository, a pull request, a deployment, etc).

You can have multiple workflows by project, each one responding to a different set of events.

In our example, we will have two workflows. The "Build" or "Main" workflow which will be triggered when there is a push the master branch or when a PR is created and the "Release" workflow which will run when a new tag is pushed to GitHub, that will create a new release of the application.

Each Workflow is composed of one or more Jobs. Our "Build" Workflow will have 3 Jobs (Lint, Build and Test) and our "Release" workflow will have a single "release" job.

Each job is made of steps. For Example, the "Unit Test" job will have steps for checkout the source code, run the tests and generating code coverage report.

The best part is that you don't have to reinvent the wheel and you can reuse existing actions built by GitHub itself or the community and even just regular Docker images in your steps.

We will see examples of all of them in the article.

Workflows are defined in YAML files located in .github/workflows directory of your repository.

Each file in the directory represents a different Workflow.

Here is how our Build workflow looks like:

name: Build and Test
on:
  push:
    branches:
      - master
  pull_request:

jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - name: Set up Go
        uses: actions/setup-go@v1
        with:
          go-version: 1.12

      - name: Check out code
        uses: actions/checkout@v1

      - name: Lint Go Code
        run: |
          export PATH=$PATH:$(go env GOPATH)/bin # temporary fix. See https://github.com/actions/setup-go/issues/14
          go get -u golang.org/x/lint/golint 
          make lint

  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Set up Go
        uses: actions/setup-go@v1
        with:
          go-version: 1.12

      - name: Check out code
        uses: actions/checkout@v1

      - name: Run Unit tests.
        run: make test-coverage

      - name: Upload Coverage report to CodeCov
        uses: codecov/codecov-action@v1.0.0
        with:
          token: ${{secrets.CODECOV_TOKEN}}
          file: ./coverage.txt

  build:
    name: Build
    runs-on: ubuntu-latest 
    needs: [lint, test]
    steps:
      - name: Set up Go
        uses: actions/setup-go@v1
        with:
          go-version: 1.12

      - name: Check out code
        uses: actions/checkout@v1

      - name: Build
        run: make build

We start by defining a name for the workflow and when it will be run.
In our case, we want it to run when there is a push to master or a pull request. There are many events you can listen to. You can read more about it here

The workflow contains 3 jobs, "lint", "test" and "build".

Let´s give a quick look at the "lint" job:

  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - name: Set up Go
        uses: actions/setup-go@v1
        with:
          go-version: 1.12

      - name: Check out code
        uses: actions/checkout@v1

      - name: Lint Go Code
        run: |
          export PATH=$PATH:$(go env GOPATH)/bin # temporary fix. See https://github.com/actions/setup-go/issues/14
          go get -u golang.org/x/lint/golint 
          make lint

Here, we specify that we want this job to run on an ubuntu machine. ("runs-on" keyword).

Actions have support for Linux, Mac, and Windows as well as Docker. In the future, it will be possible to use your own machines also as runners.

Then, we define the steps that compose our job.

First thing is to install Go. GitHub already provides an action for it, so let's use it:

 - name: Set up Go
   uses: actions/setup-go@v1
   with:
     go-version: 1.12

I think the syntax is pretty explanatory. The with keyword allows us to specify the arguments required by the action. In this case, the "setup-go" action allows us to specify the go version to use.

Next step is to check-out the source code. Again we will use a built-in action:

 - name: Check out code
   uses: actions/checkout@v1

And finally we will install and run golint tool:

- name: Lint Go Code
  run: |
    export PATH=$PATH:$(go env GOPATH)/bin
    go get -u golang.org/x/lint/golint 
    make lint

And that´s it. The rest of the jobs are pretty similar. Let´s take a look to the "Test" job.

  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Set up Go
        uses: actions/setup-go@v1
        with:
          go-version: 1.12

      - name: Check out code
        uses: actions/checkout@v1

      - name: Run Unit tests.
        run: make test-coverage

      - name: Upload Coverage report to CodeCov
        uses: codecov/codecov-action@v1.0.0
        with:
          token: ${{secrets.CODECOV_TOKEN}}
          file: ./coverage.txt

The only new thing here is that we are using a third-party action, in this case, to publish the test coverage report to CodeCov.

The usage is exactly the same as built-in actions. In here we are also using a new GitHub functionality "secrets" to store our "Codecov token" required by the CodeCov action. You can configure your secrets by accessing to your project settings -> secrets tab.

You can create your own actions in any language (Just add a Dockerfile) or if you like Typescript you can use their actions toolkit.

And with this, we finished our first workflow ;)

Let´s create a new branch and do a code change to see the PR workflow in action.

git checkout -b greet-devto

Now change our "Greet" function to "greet" also Dev.to users:

func Greet() string {
    return "Hello GitHub Actions. Dev.to is awesome"
}

We also need to update the respective unit test accordingly:

func TestGreetsGitHub(t *testing.T) {
    result := Greet()
    if result != "ello GitHub Actions. Dev.to is awesome" {
        t.Errorf("Greet() = %s; want ello GitHub Actions. Dev.to is awesome", result)
    }
}

Now push the branch and create a Pull Request to the master branch. The "Build" workflow will start immediately.

The merge will be blocked until the workflow passes and you will be able to see the status directly in the Pull Request:

Remember we have added Codecov integration? With one line of code in the workflow, we get full integration with Codecov with PR status checks and Coverage report as a PR comment:

The release workflow

It´s time to create our "Release" Workflow". Each workflow is a separate file, so we will create .github/workflows/release.yml with the following contents:

name: Release
on:
  create:
    tags:
      - v*

jobs:
  release:
    name: Release on GitHub
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v1

      - name: Validates GO releaser config
        uses: docker://goreleaser/goreleaser:latest
        with:
          args: check

      - name: Create release on GitHub
        uses: docker://goreleaser/goreleaser:latest
        with:
          args: release
        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

We specify that we only want to trigger it on newly created tags and we define a "release" job.

Note: The on condition seems to have some issues. For example, when I push a tag it´s also running the build workflow. Remember Actions is still beta, so take that into account.

The job will check-out the code and use GoReleaser official docker image to do all the work.

When using docker it´s possible to define the "args" and the "entrypoint" of the container. In this case, we will use the default entrypoint, but define a different argument on the "Validate" and "Create Release" steps.

We also specify the GITHUB_TOKEN environment variable required by Go Releaser to create our release on GitHub. This variable will be passed to the container. Note that secrets.GITHUB_TOKEN variable is injected automatically by the Actions platform, so no need to create it ourselves.

If you create a tag and push to the repository:

git tag v0.1.0
git push --tags

A new release will be created on GitHub with the application artifacts and Changelog generated by Go Releaser tool.

And we have our first pipeline built with GitHub actions. ;)

Very basic example, but I think enough to give you a good idea about how it works.

Conclusion

I am very impressed with the way GitHub Actions work right now and looking forward to the final release.

I believe GitLab is still superior for more advanced use cases as it supports for example manual approvals and parametrized builds, essential features for the enterprise, but we can´t forget that GitHub Actions is still in beta and I guess these features will come sooner or later.

Actions also have some nice features that GitLab doesn't, as Matrix builds.

The features that Actions have currently, should be more than enough for 90% of projects and I believe will be huge for Open source and personal projects.

And with all the GitHub community building all kinds of open-source actions, we can expect amazing things.

With the 3 major Git hosting providers supporting CI Pipelines, with Jenkins still very popular in enterprise and some new more specialized tools like Codefresh, I am curious about the future of more traditional CI only platforms like Travis or Circle CI.

Exciting times in this space for sure.

Thanks for reading and please if you had the chance to try the beta, give your feedback in the comment section.

Reference

Setup a local DNS server for your projects on Linux with Dnsmasq

Bruno Paz — Fri, 16 Aug 2019 19:27:34 +0000

In this article, I will explain how to setup a local DNS server on Linux.

I will use Dnsmasq, a lightweight DNS server which comes pre-installed in Operating Systems like Ubuntu or Fedora.

All the instructions on this article are based on a Fedora 30 installation. The concepts will be the same for any other *Nix based OS, but the install commands or locations of some files might differ a little.

Why setup your own DNS server?

There are many reasons why you might want to setup your local DNS server
like Caching, but for me, one of the most useful reasons is to resolve local development domains automatically to your localhost.

For simple projects, using "localhost" and a "port", will probably be enough, but for more complex projects where you need to have multiple applications running at the same time (Ex: A Microservices oriented project), it will start to become more complicated to remember all the ports and to avoid conflicts, so a domain-based approach is easier.

There are also some cases where your business logic depends on the domain of the application.

An application where each user logs-in via a subdomain or when you have different domains per country and the content is loaded based on that domain for example.

In these cases, it's essential to have a "real" domain on your local environment to mimic the expected behavior in production.

A common approach to this is to add your domains to the hosts file of your system.
While this works fine, it is not very scalable as you need to add each new domain manually to the file and it doesn't support wildcard subdomains for example.

A local DNS server can make this a one-time setup.

Installing and configuring Dnsmasq

Dnsmasq comes pre-installed in Ubuntu and Fedora. In Ubuntu is not really dnsmasq but dnsmasq-base, a stripped version of DNSmasq that is tightly integrated with the system Network manager. that is enough for our needs.

For other OSes you might need to check the respective OS documentation, but if not already installed, you should be able to install it using the OS default package manager.

After installing Dnsmasq you need to enable "NetworkManager" to use it as the default DNS server.

Create a new file /etc/NetworkManager/conf.d/dnsmasq.conf and add the following content:

[main]
dns=dnsmasq

Restart the "NetworkManager" service to persist your changes.

sudo systemctl restart NetworkManager

Now check the /etc/resolv.conf and check if there is an entry for nameserver 127.0.0.1.

Add your local domains to Dnsmasq

The next step is to configure your DNS zones for your domains.

You can specify as many domains as you want, but personally, I am using "lh" (localhost) TLD for all my local domains.

Go to /etc/NetworkManager/dnsmasq.d and create a new file with the name of your TLD, in my case lh.conf with the following contents:

address=/lh/127.0.0.1

Alternatively, if you have installed full Dnsmasq and not using the one that comes with NetworkManager, the configuration should be added to /etc/dnsmasq.d/.

This instructs Dnsmasq to resolve any query to the lh TLD to your local machine.
You don't have to always point to 127.0.0.1 tough. If you are using VMs, for example, you cant point to your VM IP address.

A good example of this use case is when using minikube.

I have created an mkube entry in Dnsmasq to resolve to the Minikube VM IP Address, so I can create an app1.mkube, app2.mkube and they will reach Minikube automatically.

Finally, you need to restart the services to reload your changes:

sudo systemctl restart dnsmasq NetworkManager

To check everything is working, open your terminal and type dig test.lh.

You should see that it resolves to 127.0.0.1. You can test any other subdomains and they all should resolve to your machine.

And voilá, you have setup a very simple DNS Server. No need to worry about ports or host file again. Any *.lh request will resolve to your machine.

Next steps

The next steps would be setup an HTTP server at your localhost to handle your requests and proxy to the correct application based on the domain name.

If you are using Docker, I recommend giving Traefik a try.

Install Traefik on your machine listening on port 80. Traefik will listen to Docker events and automatically reload its configuration when a container status changes.

You can set a hostname for the application and Traefik will do the necessary mapping, so for example, my-app.docker.lh will be proxied automatically to the correct internal port of the container.

You might need to setup some labels on your container, nothing more. Traefik and Dnsmasq will handle the rest.

This setup is out of the scope of this article. I might write another article about this.

Note for Ubuntu users

If you are using Ubuntu please check this question on Stack Overflow.

Conclusion

For smaller or single application projects, a port-based approach is perfectly fine and there is no reason to not stick with it.

But if you work on many projects at the same time, projects with complex microservices architecture, or when your application have a logic based on the domain or subdomain, Dnsmasq provides a very simple way of managing your domains without having to manually deal with the hosts file.

Reference

Using the NetworkManager's DNSMasq plugin - Fedora Magazine

What is your´s worst experience with recruiters?

Bruno Paz — Sun, 07 Jul 2019 11:07:27 +0000

If you work in tech and have a Linkedin account, the probability that you receive daily messages from recruiters is pretty high.

While a few might be interesting and realistic proposals, I would say most of them could be considered close to SPAM and show a lack of preparation and understanding of the field. (Like saying that are looking for Java devs, for someone who has Javascript in their profile).

This morning, I was reading my Twitter feed and saw this tweet:

Daniele Bottillo

@dbottillo

Recruiter also in GitHub opening PRs on my personal open source project, that's next level for me.

13:55 PM - 04 Jul 2019

A guy who was contacted by a recruiter via a Pull Request in one of his Open source repos!

It seems that recruiters are getting more and more "creative" and start using all the possible means just to get your attention!

While I understand creativity is important to get noticed in a very crowded field like tech, some actions might be too much invasive.

This inspired me to write this post and to talk about which was probably my worst experience ever with a recruiter.

Some years ago, It was another normal working day at my company when the receptionist came to me saying I had a phone call from someone saying claiming to be from the University where I graduated some years ago.

I was very surprised but went to the phone.

When I pick it up, the guy starts talking that he had an opportunity for me and if I want to schedule an interview and so on.

I ended up not understanding If he really worked at the university or not, but I believe it was a just bait to get my attention. He probably got that info as well as the company info by looking at my Linkedin profile.

He also gave me a name that I couldn't find anywhere on Linkedin, so I believe it was fake too.

And by his voice, seemed very anxious and not really secure of himself.

What can lead to someone to have this kind of behavior? Calling to your work with a bunch of lies just to grab your attention?

I believe there are good and professional recruiters out there and that they have an important role, but I think the image of the field is getting very damaged by examples like these.

So, What are your experiences with recruiters?

If you work in the recruiting field and are reading this, please also feel free to share your opinion on the topic.

Ensure the quality of your Docker Images with these cool tools

Bruno Paz — Mon, 01 Jul 2019 20:32:10 +0000

When we talk about software testing, most of the time it's about testing the application code in order to guarantee it safeties our defined code quality metrics and functional/business requirements.

The same concepts like Unit Testing or Integration Testing can also be applied for infrastructure related code.

Testing VMs and Bare metal systems can be complex and very time and resource consuming, but with containers that process is much easier.

Next, I will present some tools that you can use to ensure the quality of your Docker images.

Hadolint

Hadolint is a linter for Dockerfiles.

It can be used to ensure that you Dockerfile doesn't contain structural errors and follows the official best practices.

It also uses ShellCheck under the hood, which is a static analysis tool for shell scripts, to validate the shell code present in your Dockerfile "RUN" instructions, which is pretty neat.

Having a well written Dockerfile is the first step to a quality Image.

Container Structure Test

Container structure test is a tool developed by Google to validate the structure of your image. It can be seen as a Unit Test framework for containers.

These tests can be used to check the output of commands in an image, as well as verify metadata and contents of the filesystem.

You define your tests in a YAML file and then run the tool against an existing image locally or it some registry.

A simple metadata test, that checks the existing of environment variables, labels, ports and other can be defined as such:

metadataTest:
  env:
    - key: foo
      value: baz
  labels:
    - key: 'com.example.vendor'
      value: 'ACME Incorporated'
  exposedPorts: ["8080", "2345"]
  volumes: ["/test"]
  entrypoint: []
  cmd: ["/bin/bash"]
  workdir: "/app"

Goss and DGoss

Goss is another YAML based tool for validating a server’s configuration. It's built in Go and can be used to test all kinds of systems from Virtual Machines to containers. Dgoss its a wrapper that facilitates the usage of Goss with containers.

Here is an example of a test definition using Goss:

port:
  tcp:22:
    listening: true
    ip:
    - 0.0.0.0
  tcp6:22:
    listening: true
    ip:
    - '::'
service:
  sshd:
    enabled: true
    running: true
user:
  sshd:
    exists: true
    uid: 74
    gid: 74
    groups:
    - sshd
    home: /var/empty/sshd
    shell: /sbin/nologin
group:
  sshd:
    exists: true
    gid: 74
process:
  sshd:
    running: true

As you can see its really simple to understand.

It is very similar to "Container structure test" but with a bigger set of functions that allow you to verify stuff like services, users, packages, groups, and even HTTP endpoints. You can probably do that by writing your own commands, using Container Structure test, but goss provides these out of the box.

I think both are good options. I haven't tried Goss yet in practice. Experiment and see what you like more ;)

Kitchen CI

One of the popular tools to test infrastructure code is Kitchen CI.

Kitchen provides a test harness to execute infrastructure code on one or more platforms in isolation.

A driver plugin architecture is used to run code on various cloud providers and virtualization technologies such as Vagrant, Amazon EC2, and Docker.

It supports many testing frameworks like InSpec or Serverspec.

It's Ruby based and a lot more complex than Goss or Structure Test. It also requires many more dependencies. Nevertheless its a very powerful tool that you might want to check out.

Security Testing with Clair

A very important aspect that sometimes is forgotten when working with containers is security. Docker images still have an OS like Ubuntu or Alpine under the hood which might have software packages with known security vulnerabilities that need to be monitored and patched.

Clair is an open source project that helps to find these vulnerabilities in your docker images.

It contains a database that is updated at regular intervals so it can find the most recently discovered issues.

You can easily integrate it into your CI Pipeline to be notified on any vulnerability.

Recently I also discovered Anchore which is very similar but that besides its open source solution also provides an Enterprise solution with Dashboards and other things.

Keep your image sizes in control with Dive

Having small and optimized images is very important in order to have faster builds.

Dive is a very cool tool that allows you to explore a docker image, see its layers contents, and more.

It can help a lot to understand how your image layers are organized and find ways to shrink the size of your final image.

Even cooler, is that you can run it on your CI system and configure it to fail the build based on some metrics related to the size of the image.

Conclusion

Testing your infrastructure code besides your application code is very important to avoid unexpected problems.

In more traditional systems it´s not always easy and it can have a high cost in terms of time and resources.

With Docker and these tools, it's really simple and fast to implement some basic tests to guarantee that the required packages are installed, the specified ports are listening and the needed services are running.

Having these tests can give you extra confidence that the system as a whole will work as expected.

Do you know any other cool tool that I haven't mentioned in the article? Feel free to share.

Thanks for reading.