DEV Community: Sergio Díaz

A Little Bit of Action with AWS KMS

Sergio Díaz — Tue, 29 Sep 2020 01:03:26 +0000

TL;DR

After reading this blog post, you will have an overview on how to implement Encryption as a Service using AWS KMS.

Overview

In the previous post, we used Hashicorp Vault to perform encryption and decryption operations. Now we are going to have a look to a different approach. Why? Although Vault is an open-source solution backed by a great community. It might not be for you. For example, you need to manage it and ensure that it will have 100% uptime. Failing to do so could jeopardize your application's availability. This will also require engineering resources, so depending on priorities of your company, you might consider different options. With AWS KMS, you will be able to worry less about managing a business critical service while obtaining the advantages of Encryption as a Service.

Quick Start

The following diagram gives a general overview of how this implementation works:

Encryptor

This service identifies a user. Then, encrypts a message that is received via console prompt. When we encrypt a message, the encryptor receives a cipher-text blob (bytes) from KMS. However, the encryptor base64-encodes it before returning the message to the client. This is done to transport the data in an easier way.

Decryptor

This service identifies a user. Then, decrypts an encrypted message (which is base64-encoded) that is received via console prompt.

AWS KMS

This service will manage encryption and decryption for us. But first we need to create it. So let’s use CloudFormation (CF) to request the resources we need.

Requesting the resources in AWS using CF

You can fin the source code here. By using the repo’s CF template, we will create the following resources:

theKey (CMK)

A logical key that represents the top of a key hierarchy. We use it to perform encryption and decryption operations. By default is a symmetric key, so it can do both: encrypt and decrypt.

theAlias

This is the alias for our key. This is one is helpful to reference a key using a human friendly format. For example, You can use alias/the-alias instead of a UUID. Once an alias has been associated with theKey, the alias can be used in place of the ARN.

iCanEncryptStuffGroup and iCanDecryptStuffGroup

These IAM groups have a policy to allow KMS to encrypt or decrypt respectively.

EncryptorUser and DecryptorUser

These users are able to encrypt or decrypt according to the group they belong to.

EncryptorUserAccessKey and DecryptorUserAccessKey

These credentials allow the associated user to use the AWS API. Normally, we would create an IAM role and attach it to the EC2 instance that runs our application, but we are using local scripts [0].

Deployment

Now it’s time to deploy our infrastructure. Let’s run ./deploy.sh and you should be able to create the resources. Just make sure you update the variables accordingly and to have an AWS account with enough privileges to run the CF template.

If you see an output like the following, we are ready to go.

{
"ExportingStackId": "arn:aws:cloudformation:us-east-2:XXX:stack/kms-poc/a59778da-fe1b-11ea-adc1-0242ac120002",
"Name": "DecryptorUserAccessKey",
"Value": "AKIAXXXXXXXXXXXXXXXX"
},
{
"ExportingStackId": "arn:aws:cloudformation:us-east-2:XXX:stack/kms-poc/a59778da-fe1b-11ea-adc1-0242ac120002",
"Name": "DecryptorUserSecretAccessKey",
"Value": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
{
"ExportingStackId": "arn:aws:cloudformation:us-east-2:XXX:stack/kms-poc/a59778da-fe1b-11ea-adc1-0242ac120002",
"Name": "EncryptorUserAccessKey",
"Value": "AKIAXXXXXXXXXXXXXXXX"
},
{

"ExportingStackId": "arn:aws:cloudformation:us-east-2:XXX:stack/kms-poc/a59778da-fe1b-11ea-adc1-0242ac120002",
"Name": "EncryptorUserSecretAccessKey",
"Value": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
{
"ExportingStackId": "arn:aws:cloudformation:us-east-2:XXX:stack/kms-poc/a59778da-fe1b-11ea-adc1-0242ac120002",
"Name": "theKeyAlias",
"Value": "alias/kms-poc"
},
{
"ExportingStackId": "arn:aws:cloudformation:us-east-2:XXX:stack/kms-poc/a59778da-fe1b-11ea-adc1-0242ac120002",
"Name": "theKeyId",
"Value": "a59778da-fe1b-11ea-adc1-0242ac120002"
}

Create a .env file and use env.example for some inspiration. You should be able to get all the info you need from the previous output.

Note: For the KEY_ID env variable’s value, we can use either theKeyId’s or the theKeyAlias’s value.

How to Encrypt

Run python3 encryptor.py
Use either alice or bob to “authenticate”
Write a message to encrypt
Hit Enter

You should see a base64 output like this:

AQICAHhQUp2GIUg1Tf/r2I9nmNWsIJGDrRYieOviWRC0SLz/bwEXoiPmO1CgXGPhO5su0nhJAAAAbjBsBgkqhkiG9w0BBwagXzBdAgEAMFgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9eb1WUEj8kK/WBjvAgEQgCs9K6dPBPgWZ+ZqKqXdIt1S/CUE1Xj9fcUq9vo95Mw/6XKv8yQkciWBnH55

How to Decrypt

Copy the output from the encryptor and proceed to run the decryptor by doing: python3 decryptor.py
Use the same user you used to encrypt earlier during the process (alice or bob)
Paste the base64 output
If everything went as planned, you should see something like:

Username:

alice

Hello alice, please enter your message to decrypt:

AQICAHhQUp2GIUg1Tf/r2I9nmNWsIJGDrRYieOviWRC0SLz/bwEXoiPmO1CgXGPhO5su0nhJAAAAbjBsBgkqhkiG9w0BBwagXzBdAgEAMFgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9eb1WUEj8kK/WBjvAgEQgCs9K6dPBPgWZ+ZqKqXdIt1S/CUE1Xj9fcUq9vo95Mw/6XKv8yQkciWBnH55

Your decrypted message: this is a secret

Encryption Context

Now try encrypting a message as alice, but decrypting the message as bob. Even though we are using the same key and we have the appropriate permissions, we get an InvalidCiphertextException. This happens because we use a different EncryptionContext while encrypting and decrypting. Even though this parameter is optional, you should use it. This value can help us understand why a given master key was used and which operation was done. This will be handy when we are looking to the logs.

While the EncryptionContext doesn’t need to be kept as a secret, it helps to know that you are encrypting and decrypting between services in a “conscious” way. For example, if your application is handling messages for alice, but by mistake receives a message from bob, the decryption operation will fail. With that being said, the Encryption + EncryptionContext would be like having both: belt and suspenders.

Key Rotation

When using KMS this functionality can be seen as an advantage or disadvantage. For example, if you use EnableKeyRotation while creating a AWS::KMS::Key, KMS automatically creates a new key material for it, and rotates it every year. If for some reason you need to do it more often, you would need to do a manual rotation. Another thing to add is that KMS retains all the key versions until you delete it. In other words, you cannot delete an old version of the CMK.

Other things to Consider

Key Strategy

For this example we only used one key to encrypt and decrypt the user's messages. What would happen if we have millions of users that are encrypting millions of messages? In this case, you will need to adopt a Key Hierarchy Model that meets the needs for your organization. If you want to know more about this, I suggest starting here.

Vendor Lock-in

While AWS API is straight-forward, it would be interesting to know how you could migrate to another Key Management System without having the problem to re-encrypt every single blob that you have saved so far.

Disaster Recovery (DR)

If you can't get access to the crypto keys, you cannot do what your application does. So it might be worth looking on how to implement a multi-region strategy in case of an outage.

Wrapping it Up

As you could see, setting up and performing basic operations with AWS KMS is straight forward. Another advantage is that you will not worry about the service's uptime. This is great, especially if your business relies on those encryption and decryption operations. Nonetheless, you still need to put thought into things such as defining an adequate encryption context, coming up with a key strategy, analyzing how you could avoid vendor lock-in, and placing a disaster recovery strategy.

Reference

[0]https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html

Encryption as a Service in Action

Sergio Díaz — Tue, 15 Sep 2020 13:22:31 +0000

TL;DR

After reading this blog post you will have an overview of why we need encryption as a service. Also, we will implement a PoC using Python and Hashicorp Vault to apply what we just learned.

Overview

Nowadays there are companies that deal with sensitive data such as banking or health information. This means that in the case of a security breach, the company could have legal, financial, and bad PR repercussions. As a result, there are some industry standard guidelines for storing this type of data (eg. Payment Card Industry Data Security Standard). For example, when it comes down to store Credit Card Data, organizations are required to encrypt the account numbers stored in databases.

Encryption in transit

While encryption at rest can be seamlessly be done by using a cloud provider, handling the data when it’s moving around services can be tricky. Yes, you can use a secure communication channel (eg. TLS) between services, but this is not enough. One thing that could go wrong, even when using TLS, is logging sensitive information by mistake. This can happen even to companies that are worth billions of dollars and with an awesome engineering team.

Handling encryption at the application level

This is possible since there are good cryptographic libraries out there. However, developers should do the implementation in a correct way (eg. not using deprecated ciphers).

Furthermore, while a company might be running a Majestic Monolith, chances are that you have other services that interact with your main service. This leads to questions such as:

What happens if more than one service needs access to the sensitive data?
Which service will handle encryption and/or decryption?
How will encryption keys be handled, rotated, and distributed?
What happens if a key is leaked?

Using a Key Management System

The solution is to protect sensitive data with a centralized key management system. This can be done using Hashicorp Vault, AWS KMS, Google CMK, etc. The idea is to delegate the responsibility of encryption and decryption to this service. Using a solution like Hashicorp Vault, allows you to encrypt and decrypt application data with an HTTPS API call. This means that:

Data can be encrypted at rest
Data is secured in Transit (TLS)
Key handling and cryptographic implementations is taken care by Vault, not by developers
More services could be added to interact with the sensitive data

How it works?

You can find the source code here.

As you can see we have 3 services:

API

This application allows you to create a "credit card" with a name and a PAN (primary account number). All data is stored encrypted and API doesn't know anything about the type encryption used or how to decrypt it.

Card Processor

This service process the credit cards we have created so far according to the business logic. Just as with the API, it doesn't know anything about encryption or decryption.

Vault

This service handles the encryption done by the API and the decryption done by the Card Processor Service. If a new service is added in the future, everything regarding encryption or decryption would still depend 100% on Vault. The best part is that no changes would be made in the other services.

How to Run

Just do docker-compose up --build and you should be ready to go.

Configuration

Run the following script:

./bin/vault.sh

3 things just happened:

The transit Secrets Engine was enabled. This tells vault to not save data and turn on the encryption as a service functionality.
The root token was created. This one is used by our apps to encrypt and decrypt.
A symmetric key was generated. This one is used to encrypt and decrypt the respective data.

Grab the token from the output value and put it as VAULT_TOKEN env var in:

app/settings.py
card_processor.py

Using the credit card API

Send a POST request to http://localhost:8000/credit-cards/ with the following body:

{
    "name":"Carlos Gardel",
    "pan":"4539296620131157"
}

If everything went well you should receive the the encrypted Primary Account Number (pan).

{
    "name": "Carlos Gardel",
    "pan": "vault:v1:JnC8pS/zmHhHPGd7dk5eCGilnUi8odvRIBP9Z+rBmMLAWXJ/dgYqGAU4MTk="
}

If you do a GET request to http://localhost:8000/credit-cards/ you will still see the encrypted PAN field.

Using the card processor service

Now let's processes the card using the card_processor.py service.

python3 card_processor.py

We should be able to see that the card was successfully decrypted and processed:

Processing CARLOS GARDEL card with number 4539296620131157

Rotating Keys

To minimize the risk if a key is leaked, let's tell Vault to rotate it.

./bin/rotate.sh

If we do another POST request to the credit card application, we will see that we used v2 to perform the encryption.

{
    "name": "Aníbal Troilo",
    "pan": "vault:v2:vdIlXggLzrM4n5Xlzxh6a/xpmd7yz/F9MsoifuR/kmOodGKV5wPaWvMMiEw="
}

Let's do a GET and see our different encryption key version (v1 and v2):

[
    {
        "name": "Carlos Gardel",
        "pan": "vault:v1:JnC8pS/zmHhHPGd7dk5eCGilnUi8odvRIBP9Z+rBmMLAWXJ/dgYqGAU4MTk="
    },
    {
        "name": "Aníbal Troilo",
        "pan": "vault:v2:vdIlXggLzrM4n5Xlzxh6a/xpmd7yz/F9MsoifuR/kmOodGKV5wPaWvMMiEw="
    }
]

Let's go back to the card_processor.py service. Now we are able to decrypt and process both entries. One was decrypted with v1 and the other one with v2.

Processing CARLOS GARDEL card with number 4539296620131157
Processing ANÍBAL TROILO card with number 2720997130887021

Voilà. Encryption keys and rotation were handled by Vault.

Further considerations

Are we production ready yet?

I don't think so.

We are using a root token
TLS and communication between Vault and services is yet to be implemented
Vault needs to be sealed
We need to make sure that the Vault service is reliable (did someone say K8s?)

Open Source vs Cloud Provider Solution

Do you want to deal with a self-hosted open source solution or would you rather deal with your respective cloud provider’s solution?

Wrapping It Up

If you want to discuss or challenge the implementation I’m happy to do so. Just leave a comment or find me on twitter.

References:

Deploying a Bastion Host in AWS using CloudFormation

Sergio Díaz — Tue, 21 Apr 2020 13:58:02 +0000

Overview

In this blog post, we are going to talk about what is Bastion Host and why do we need one. Afterward, we are going to deploy a proof of concept using AWS CloudFormation.

Bastion Who?

Although toil is highly discouraged, sometimes we need to ssh into an instance in order to do some kind of debugging. As a result, we need to expose that instance to the whole internet and that is no bueno. One way to prevent this from happening is to implement a Bastion Host.

A bastion host is a server whose purpose is to provide access to a private network from an external network, such as the Internet.

Why do we need one?

The idea of implementing this is being able to reduce the attack surface of our

infrastructure by doing 2 things:

Removing the application instances (could also be a database instance) or other servers that are not meant to be open to the world.
Being able to harden one machine (the bastion) and not each and every other server in our infrastructure. So, in this case, the m̶o̶r̶e̶ less the merrier.

Another benefit that the Bastion Host can have is logging in order to prevent repudiation. This works because engineers have their own key pair. As a result, you can keep track of what Alice and Bob did during their last session.

What are we going to deploy?

The idea is that our on-call engineer will ssh her way into the App Instance via Bastion Host. In order to replicate this setup, we need to deploy 15+ AWS resources, but let’s focus on the ones that are in the diagram:

VPC

We need one so we can create the virtual network where our instances will run

Private Subnet

We need a network that can only receive internal traffic (we only need a private IP address)

Public Subnet

We need a network that can receive traffic from the Internet (we need a public IP address)

Bastion Security Group (SG)

We need it to make sure the Bastion Host Instance can receive traffic from port 22 (SSH).

Application SG

We need to make sure our App Instance can receive traffic from our Bastion Host SG.

Bastion Host (EC2)

We need a server that we can use as a Bastion Host

App Instance (EC2)

We need a server that is not exposed to the internet

Getting Started

You can find the relevant files in GitHub.

Prerequisites

Make sure you have an AWS account
Make sure you have a user with the appropriate roles
Create a key pair in the us-east-1 availability zone. We will use the keys to connect to our instance.

main.yml

We can divide this file into 3 sections:

Parameters: Where we import variables from deploy.sh (more about it coming next) so we can use them with our resources’ attributes.
Resources: Where we define all the AWS resources that we need for this setup.
Output: If everything goes according to plan, we want to import the IP addresses from our created instances.

Remember: Although this is a simple setup, we need at least 15 AWS resources to make the desired implementation work. For example, we need an Internet Gateway so our Bastion Instance can talk to the internet and we need a Route Table to direct network traffic.

Set up the vars

# deploy.sh
STACK_NAME=bastion-poc  
REGION=us-east-1  
CLI_PROFILE=<your-aws-profile-with-an-appropiate-role>  
EC2_INSTANCE_TYPE=t2.micro  
KEY_NAME=<your-key-pair-name>

Run the deployment script

In this script, we set up our credentials and we run a command to deploy the main.yml template to AWS. If everything goes well, you should expect 2 IP addresses: One from the Bastion Instance (public) and one from the App instance (private).

Go to your terminal and run the following:

./deploy.sh

Note: If you want to debug or see what happened, go to the respective

CloudFormation stack in the AWS console.

Config your ssh config file

Now that we have our implementation we are ready to pray to the demo gods and test our implementation. But before ssh’ing anywhere, we need to do one more thing.

Go to ~/.ssh/config and add the following hosts:

...

### The Bastion Host  
Host bastion-host-poc  
 HostName <public-ip-from-output>  
 User ec2-user  
 Port 22  
 IdentityFile ~/.ssh/<your-key-pair-private-key>\### The App Host  

### The App Host  
Host app-host-poc  
 HostName <private-ip-from-output>  
 User ec2-user  
 IdentityFile ~/.ssh/<your-key-pair-private-key>  
 ProxyJump bastion-host-poc

SSH’ing your way in

If everything went well (and if we prayed to the demo gods) we should be able to
ssh to the App Instance.

Go to your terminal and ssh into it:

ssh app-host-poc

Voilà. You are inside a machine that is running in a private subnet. Isn’t it

cool?

Wrapping it up

Remember, this is just a Proof of Concept. For example, the Application Instance
can still send traffic to the whole world (do you really want that?). Similarly,
the Bastion Instance has yet to be hardened.

Implementing a Bastion can be useful for your current processes, especially if

you have some instances exposed to the world and/or you want to control

who can ssh into your infrastructure.

Although you probably have a more sophisticated setup, a Bastion Host might be
the right solution for you and this could be the kickstart of your
implementation.

Also published on Medium.

Resources

Amazon EC2 key pairs and Linux instances

How to Record SSH Sessions Established Through a Bastion Host

Diagrams were made using: https://www.planttext.com/

How I Went From Localhost To Production Using The Devops Way Part 1 Of N

Sergio Díaz — Sat, 14 Mar 2020 16:54:03 +0000

Photo by Robert Baker on Unsplash

This post covers:

Making a case for automation
Defining what is a Pipeline
Defining a game plan to automatically ship an application to a container repository
Setting up a Continuous Integration Platform
Breaking down the implementation

Making a Case for Automation

Building a compiler is not an easy task. It involves the development of many modules such as a lexer, parser and a virtual machine. It also involves the processing of variable declarations (e.g. int a = 0) and the evaluation of simple arithmetic expressions ( e.g. a = 4 * 10 + 2) all the way to calling returning functions with parameter such as factorial recursive function:

Along the process we found out a lot of things during the development phase that were not considered during the design phase (I swear I’ve never heard of this problem before). As a result, we were constantly moving the grammar to make sure edge cases worked. That is why we decided to take a step back and take advantage of the unittest module that python3 provides. For each .tl file, our programming language file extension, we created a test. This allowed us to make changes faster while eliminating the fear of breaking previous working code.

Tests were focused on three different categories:

Correctness of the intermediate code generation (quadruples)
Expected failures of .tl files with the expected number of errors
Correctness in expected executed output

As a result, doing make test (Viva la Makefile) runs 50+ tests. But what happens if a developer forgets to run the tests before doing a pull request or merging to master? Thankfully we use GitHub, so going back to a specific commit is possible, but is it practical? Do we really want to go through that painful way?

Furthermore, trendlit, our compiler, is meant to be cloud-based. So that means that we also need an easy way to deploy it. On top of that, since trendlit runs inside a Docker container, we also need a way to build the Docker image, check if the current version already exists in the container repository (e.g. Docker Hub), and eventually push that image to the registry. Therefore, we decided to do it the DevOps way.

But First, What is a Pipeline?

Photo by tian kuan on Unsplash

A pipeline is a set of automated processes that allow Developers and DevOps professionals to reliably and efficiently compile, build and deploy their code to their production compute platforms. There is no hard and fast rule stating what a pipeline should like like and the tools it must utilize, however the most common components of a pipeline are; build automation/continuous integration, test automation, and deployment automation.

Implementation Roadmap

As we previously mentioned, we want to manage trendlit the DevOps way. To do this, we will implement steps of Continuous Integration (CI), Continuous Delivery (CD) and Infrastructure as a Service (IaaS). For this post, we are going to focus on the Continuous Integration part.

Why would we even want CI?

Continuous Integration is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. [5]

As a result, errors can be detected faster and can be located more easily. This allows an organization to deliver software more rapidly, while reducing the risk for each release.

For this example, the idea is that a developer, while adding or fixing a feature, can go from localhost to the application being pushed to a registry within minutes, while relying on an automated process. How are we going to achieve this? Well, you guessed it: we are going to use a pipeline. So the workflow will look something like this:

Diagram generated by PlantUML: https://bit.ly/2W1D6OR

A developer adds new feature to the application.
The application goes through an automated testing process.
When the developer makes a pull request, the application goes under the four-eyes principle [4] and if it is approved by another peer and the automated tests pass, the new feature is merged to the master branch of the code repository.
A new version of the application is automatically built and packaged into a container.
The container is pushed to a container repository.

Getting Started

In order to run the implementation roadmap described above, we chose to use the following tech stack:

Code Repository

We will use GitHub in order to host the application’s source code.

CI Platform

When it comes down to pipelines, we can find them in all colors and flavors. For this project, we chose to go with CircleCI because it integrates easily with GitHub and allows SSH access to build instances, which is handy for debugging the build steps. Is worth mentioning that we could have replaced CicleCI with another automation server such as Jenkins, but it takes longer to setup because it is self-hosted.

Container Repository

We’ll use the repository that is provided by Docker Hub.

Setting up CircleCI

This is the application’s code repository: https://github.com/shekodn/trendlit-tutorial-1

Here you can also find all the scripts that the pipeline is going to run in order to test, build, check, and push the application to Docker Hub.

Steps

Fork the project, so you can create your own pipeline.
Go to hub.docker.com and create an account and a repository with trendlit-tutorial-1 as namespace.
Sign up to CircleCI and link your GitHub account.
Select the forked version of trendlit-tutorial-1 and build the project. The build will fail because you need to put your Docker Hub’s credentials. Also, you need to change the REPOSITORY variable in each bash script (or at least in docker_build.sh, docker_check.sh and docker_push.sh) inside the scripts directory to the name of your Docker Hub’s repository.
Don’t forget to push the changes to the master branch.

In your CircleCI’s dashboard in the project list go to trendlit-tutorial-1 settings located in https://circleci.com/gh/YOURUSERNAME/trendlit-tutorial-1/edit
Under Build Settings you will see Environment Variables. Click there and add DOCKER_USER and DOCKER_PASS (a.k.a. your Docker Hub’s credentials) as variables with their respective values. These variables will allow CircleCI to upload the application’s container into your Docker Hub’s repository.
Rerun the workflow and this time it should succeed.

This is the desired workflow after a Pull Request is merged to the master branch.

Afterwards, go to your Docker Hub and you should see three new tags. One with the trendlit’s RELEASE (a.k.a version) that was specified at the top of the Makefile, other one with a shortened version of the commit hash and the other one named latest. If you want to learn more about tagging docker images, you can have a look here.

Tags were automatically generated by the Pipeline

How does it work?

Webhooks

The pipeline is able to listen each time there is a change in the repository thanks to a webhook. According to GitHub, webhooks allow external services to be notified when certain events happen. When the specified events happen, GitHub sends a POST request to each of the URLs provided [3].

In other words, when you link your GitHub account with CircleCI, you allow the latter to perform some actions in your repositories. One of them, is automatically configuring a webhook. Do you remember when we built the application for the first time? Well, it was created back then. So every time there is an event such as a pushed commit or a pull request, this webhook sends a notification to CircleCI in order to trigger the build.

If you want to see how the webhook looks like, go to your project’s settings on GitHub and have a look. There you will be able to see which events trigger the webhook.

Repository/Settings/Webhooks

Decomposing the Config File

Once the build is triggered by the webhook, CircleCI goes to the config.yml file located inside the .circleci directory.

1. Jobs

The pipeline consists of 2 main jobs: Test and Deploy.

2. Tests

Here we select Docker as an executor type. Then, we use the python:3.7-alpine image in order to to run some tests.

3. Deploy

Here we use an official Docker image called docker:stable-git, because we need a Docker image that installs Docker and has git [1].

Then we use setup_remote_docker, because according to jpetazzo, we should think twice before using Docker inside Docker [2]. As a result, all the Docker commands such as docker build and docker push will be safely executed in this new environment.

As you may have noticed, Deploy has several run steps:

We run apk add make in order to get trendlit’s release version. You can get it in your terminal by running make version. The latter prints the RELEASE variable located at the top of the project’s Makefile.
Remember when we set some environmental variables? Well, there you go: We run docker login so the pipeline has permission to push the docker container to the respective container repository.
We use docker_build.sh in order to build the Docker container with the following format: REPOSITORY/IMAGE:TAG.
Afterwards, we use the docker_check script in order to check if the current release already exists in the respective container repository. Do you remember the RELEASE variable located at the top of the Makefile? Well, if the version already exists IT WILL BREAK THE PIPELINE, because you don’t want to overwrite an existing version right? You can manually fix this by editing the Makefile and assigning a different version or you can run make bump to increase the version in a smoother way ;)
Last but not least, we push the image that docker_build.sh previously built to Docker Hub.

4. Workflows

Here we defined a workflow with 2 jobs: Test and Deploy. Test will occur each time a change is made to the application in any given branch, while Deploy will always wait for the Test job to finish and will only run in the master branch. This is because we don’t want to build and ship a container to the registry each time we do a change in a feature branch.

Wrapping it up

Although we finished the Implementation Roadmap, the implementation as a whole is far from done. It is easy to look at all the missing things and considerations, but remember where we started. Now we have a fully automated process that tests our application and pushes it to a container repository without any human interaction. It’s worth pointing out that some important aspects such as security were neglected, but this is only one part of the implementation’s MVP (Continuous Delivery is coming soon).

If you have any suggestions, I would love to hear them. I’m pretty sure we can include them in future roadmaps. BTW, Pull Requests are open ;) so I would like to collaborate with you if you think we can improve the process. In the meantime, I will be working on part 2.

References

[0] https://www.bmc.com/blogs/deployment-pipeline/

[1] https://circleci.com/docs/2.0/executor-types/

[2] https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

[3] https://developer.github.com/webhooks/

[4] https://whatis.techtarget.com/definition/four-eyes-principle

[5] https://www.thoughtworks.com/continuous-integration

Also published on Medium.