DEV Community: Roland 🐺

Deploying a VM using AWS Cloud Formation

Roland 🐺 — Thu, 05 Nov 2020 04:34:08 +0000

Getting started with AWS CloudFormation by deploying a VM

Recently I started using Amazon AWS and quickly realized that provisioning virtual machines, security policies and other network configurations is done one by one, which is tedious and needs to be automated. I had two options: either create a custom script to provision every single service or learn how to use CloudFormation.

CloudFormation is a provisioning tool for AWS resources that requires a YAML/JSON formatted file inside which cloud infrastructure is described. This definitely seems like the better option, but being completely new to AWS the learning curve is steep and it is difficult to understand the terminology, the various components and how to combine them. Even the graphical user interface, the “CloudFormation Designer” is not that easy to use for new users.

Screenshot of CloudFormation Designer

What did I do? First I learned how to provision all the components manually one by one, which took more time than I would like to admit, but was essential for me to understand the components and which ones depend on another. Eventually manually provisioning resources became easier and all the different services became familiar, that allowed me to transition my set up to CloudFormation. It was a long process. Hopefully this article will allow others to use CloudFormation much sooner.

It’s difficult to cover all scenarios and edge cases, so please be prepared to dig through some AWS documentation and learn more about some of the individual components. Unfortunately, there are always some road bumps…

However, I’m confident, that providing you with my configuration example will make learning CloudFormation easier. There are a few things I will skim over for the sake of keeping this article short and to the point. Please use the comment section to provide feedback and ask questions.

Goal

The goal of this tutorial is to set up a single virtual machine and expose it to the internet using CloudFormation.

Requirements

To follow this tutorial an AWS account is required. The AWS command line interface is not needed.

AWS has a free tier, that should allow you to follow this tutorial at no cost: https://aws.amazon.com/free/

AWS services and resources

The following AWS services, also referred to as resources, are needed for a simple virtual machine (VM) setup:

Stack — A set of resources provisioned and managed by CloudFormation is referred to as a Stack.
Image — Images (AMI) describe the root volume of a virtual machine. For example they can contain the operating system or application server.
Virtual Machine — It’s basically an artificial computer. It behaves like an actual computer but does not physically exist. It exists and runs on a server. Virtual Machines come in different sizes (CPU, Memory (RAM) and other hardware components) and can be based on different images.
Storage Volume — A hard drive that is plugged into the virtual machine.
Security Group — Security group is a virtual firewall for our set up in AWS. It controls all traffic in and out of segment (VPC) of the AWS cloud.
VPC — Every computer and cloud infrastructure needs networking. To protect our instance in AWS, we create a virtual network, exclusive for our purpose, also called a Virtual Private Cloud (VPC)
Subnet — A subnet defines the range of IP addresses in our VPC
Route Table — Defines and determines traffic flow
Internet Gateway — Internet Gateways enable communication between our VPC and the internet
Elastic IP Address — An Elastic IP Address is a static IPv4 address. It can be reused and point to different VPCs.
CIDR blocks — CIDR blocks define valid IP address ranges and are needed by Subnets
Associations — CloudFormation requires associating certain components with each other. For example a Subnet and a Routing Table.
KeyPair — Amazon EC2 requires keys for accessing a virtual machine. These ssh keys are referred to as KeyPairs and often the KeyName (name of the key) is required for the creation of virtual machines. Table 1: Overview of AWS Components needed for a CloudFormation set up

The CloudFormation definition file

The CloudFormation file defines what we want to provision. It can define all the components above and much more.

I think of it as the one file to rule them all, even though you can have as many CloudFormation definition files as you want.

The definition file can be written in either JSON or YAML. I chose YAML because the notation is shorter and it’s easy to add comments. Feel free to use the notation of your choice though, the CloudFormation designer can convert between either format with a simple click.

The basic structure my template uses entails 3 major sections (there are more, but we don’t need them right now).

Parameters
Mappings
Resources

Parameters

The Parameters section allows for the definition of inputs that can be provided through the AWS interface or the AWS CLI when creating an instance. It’s optional, but quite useful to make our configurations more reusable. In this section we define parameter-name, expected value type, data value validation, description and default value. They’re especially useful once we get really comfortable with CloudFormation.

Parameters:
  KeyName:
    Default: 'EC2Key'
    Description: 'Enter name of an existing EC2 KeyPair to enable SSH access to the instance'
    Type: 'AWS::EC2::KeyPair::KeyName'
    ConstraintDescription: Name of an existing EC2 KeyPair.
  StackPostfix:
    Default: 'vm-1'
    Description: 'Postfix for this vm stack. This is purely for ease of use, to recognize a particular VM more easily.'
    Type: String
    MinLength: '1'
    MaxLength: '64'
  InstanceType:
    Description: 'WebServer EC2 instance type. Change based on application needs.'
    Type: String
    Default: t2.micro
    AllowedValues:
      - t1.micro
      - t2.nano
      - t2.micro
      - t2.small
      - t2.medium
      - t2.large
    ConstraintDescription: must be a valid EC2 instance type.
  CidrBlock:
    Description: 'Enter CIDR block that the VPC should use. Example: 10.0.0.0/16'
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 10.0.0.0/16
    AllowedPattern: '(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})'
    ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
  LimitedSSHAccess:
    Description: 'Enter CIDR for network that may access via SSH'
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: '(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})'
    ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.

In my template I’m defining 5 parameters with following default values:

The EC2 key that may be used to connect to the EC2 instance. A key is required to connect to your VM via SSH. When creating an EC2 instance manually through the web interface AWS will offers to create one for you. However, when using CloudFormation we need to create a key beforehand.
The StackPostfix, is of no functional value, it just describes a string that will be appended to all the components that this CloudFormation template will create. I find it helpful to have a consistent postfix throughout all resources when looking at them through the AWS console. It allows for quick identification, which virtual machines all resources belong to. The default value is vm-1.
The instance type defines which EC2 virtual machine instance type should be provisioned. For testing purposes it is configured to use one of the smallest available instances types: t2.micro.
The “CIDR block” describes what IP range the VPC should be using internally.
The “LimitedSSHAccess” parameter allows you to restrict which IP addresses may access the virtual machine via SSH. It’s a good idea to restrict this to the smallest possible IP range. E.g. your personal IP address. If you don’t know which IP address range to use, set it to 0.0.0.0/0, this will allow SSH access from anywhere.

Documentation: AWS Parameters, AWS Instance Types

Mappings

At its core, mappings are just plain key-value matching. We mainly use them in our configuration to map to EC2 instance types — which kind of machines we want to provision (size-wise) and what images we want to use on said instances. AWS provides a large variety of existing images ranging from Windows to Linux distributions or even allowing us to uploading our own. For this tutorial we’ll use an existing Linux based Amazon Machine Image (AMI), that is already provided and that already comes with pre-installed software.

An example of Amazon’s image marketplace. This screenshot is taken from the AWS Console, during an EC2 instance creation process.

While all images types can be viewed through the general catalog under https://aws.amazon.com/marketplace, we do need to access the EC2 instance creation page at least once, to retrieve the Image ID’s that allow Cloud Formation to understand which exact image we need for our virtual machine that we are going to create. These Image ID’s are region and instance specific. A Linux image in us-west-1 does not have the same ID in us-west-2. Therefore we need to create all the mappings, based on regions.

AWSRegionArch2AMI:
    # Note that image's are AWS region specific. Hence,
    # the "same" image has to be defined per region
    # and the image ID's will vary and additional mappings
    # may need to be added
    us-west-1: #
      # AWS image, comes with docker pre-installed
      HVMA2018: ami-09a3e40793c7092f5 
      # Ubuntu 18
      HVMU18: ami-0d705db840ec5f0c5
    us-east-1: #
      HVMA2018: ami-032930428bf1abbff
      HVMU18: ami-0ac80df6eff0e70b5

The marketplace is quite useful, it also gives you estimates on how expensive machines are in certain regions. For example review the pricing information section of this image.

Feel free to browse the AMI Marketplace for Elastic Compute Cloud (EC2) yourself and discover what other images they have to offer.

Documentation: AWS Mappings, Amazon Machine Images (AMI), AMI Marketplace

Resources

Resources are the bits and pieces we need from Amazon. It’s all the components we described earlier.

Note that in the following example, I defined resources with the names VM, FormationSecurityGroup, VMvpc, VMInternetGateway,...

These names can be customized. The information that tells CloudFormation what the desired resources are is captured in the Type field.

Resources:
  # Define an EC2 instance. This is our virtual machine
  VM:
    Type: 'AWS::EC2::Instance'
    ...

  # This defines the Security Group, used for the VM above
  FormationSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    ...

  # This defines the VPC
  VMvpc:
    Type: AWS::EC2::VPC
    ...

  # This defines the Internet Gateway
  VMInternetGateway:
    Type: AWS::EC2::InternetGateway
    ...

  # This attaches the Internet Gateway (IGW) to a VPC
  AttachGateway:
    Type: AWS::EC2::VPCGatewayAttachment
    ...

  # This defines a Subnet
  VMSubnet:
    Type: AWS::EC2::Subnet
    ...

  # This defines the Routing Tables for our subnets
  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    ...

  # This defines the routes. Public route table 
  # has direct routing to IGW
  PublicRouteVM:
    Type: AWS::EC2::Route
    ...

  # Associate Routing Table to Subnet
  SubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    ...

  # This defines the Elastic IP Address. 
  # On creation AWS will provision an
  # IP address and will then be usable for your VM
  ElasticIPAddress:
    Type: AWS::EC2::EIP
    ...

This is just a basic structure of what goes into the resource section of the configuration file. Review the entire template for more details.

Documentation: Resources Reference, Template Structure / Anatomy

Provision a virtual machine

Now that the most important components and pieces are covered, we can provision the virtual machine. The following 4 steps walk us through the creation of a set of SSH keys for AWS, how to submit the template to the CloudFormation designer and how to create a Stack with it. Lastly we will run a small web server to validate a successful deployment and cleanup the stack.

1. Create KeyPair

Go to the EC2 management and create a KeyPair. If you’re using North California (us-west-1) you may use this link and select "create key pair". For a different region, make sure to change to select the desired location in the top right menu, then go to the "EC2" service > scroll down to "Network & Security" > "Key Pairs".

Example for a new KeyPair with name VMKey-NorthCalifornia and the file format pem.

2. Open the CloudFormation designer

The CloudFormation designer can be opened from the website https://aws.amazon.com/cloudformation/ or from the AWS Console by navigating to the CloudFormation service and then selecting “Create Stack” and then “Create template in Designer”.

Opening CloudFormation designer (GIF animation)

After opening the designer, select “template” at the bottom left and then change the editor template language from JSON to YAML.

Then copy my template from GitLab into the template viewer and select refresh in the top right. The interface should then update and display something like the screenshot below:

Screenshot of CloudFormation designer, after pasting my template.

The YAML defines all the resources and associates individual components together. Feel free to browse through it and review all the components. I added comments that hopefully explain what most of the various resources do and what they are for.

Make sure to update the LimitedSSHAccess CIDR block to an IP range that works for you, before proceeding, otherwise you won't be able to access the machine later.

Example, if your IP address is 67.188.42.123 a possible CIDR block would be 67.188.0.0/16

When ready, submit the design to the CloudFormation service by selecting the “Create Stack” button in the top left.

Screenshot of the Create Stack button.

The CloudFormation stack creation process should be displayed and “template is ready” should be selected. When ready, select “Next”.

Screenshot of the Create Stack screen, now using the template we created in the CloudFormation designer.

The stack detail screen will be displayed next, asking for a name for the stack and allowing the parameters to be changed. Change them as desired and make sure to change the “KeyName” to the KeyPair that was previously created in the EC2 AWS service section.

Screenshot of the stack detail screen. This step requires a name for the CloudFormation stack to be added and allows parameters to be overridden. Make sure to update the KeyName to the KeyPair that has been created for your account for the selected AWS region.

Once the details section is complete continue selecting next until we reach the “review” screen and select “Create stack” after verifying all values.

3. Creating the CloudFormation stack

CloudFormation will now provision the EC2 instance and all required components. The events section can be manually refreshed and will contain more event entries over time as CloudFormation provisions more and more of the requested resources.

Screenshot of CloudFormation stack creation page. It shows that a stack creation just started.

Should an error occur, the entire stack creation will be aborted and any resources of the stack, that were successfully created, will be removed.

Screenshot of a potential error that may occur. In case of a problem CloudFormation will abort, automatically rollback and undo any created resources. Check the Events log for information on what happened. — The resolution for this particular problem was waiting a few hours for a verification email by Amazon and then simply attempting the stack creation again.

If everything goes well CloudFormation will eventually switch from CREATE_IN_PROGRESS to CREATE_COMPLETE.

Screenshot of CloudFormation Stack creation screen after it successfully completed. Note how on the left a message “CREATE_COMPLETE” is being displayed.

Screenshot of CloudFormation Stack “Resource” tab, listing all the provisioned resources and the corresponding status.

4. Review running VM instance

If everything works out as expected, the instance of our VM should be running and accessible via the internet. In the CloudFormation section of AWS the CloudFormationDemo stack should be successfully created and listed.

By opening the AWS Console website we can navigate to EC2 and should see a running instance of vm-1. Furthermore, under "Network & Security" > Elastic IP addresses, the public IP address with which we can reach our virtual machine can be reached, should be listed as well.

Screenshot of EC2 services overview

Navigating to the VPC services should allow us to see the definitions of the VPC, subnets, routing tables and internet gateways.

Screenshot of VPC services overview

Currently there’s not much to see when hitting the public IP address of the virtual machine. However, by SSH’ing into the machine using ssh -i {KEY_PAIR_NAME}.pem ec2-user@{IP-ADDRESS} and then starting a Python web server, it should be possible to see the following page:

#### Connect to VM
# Replace KeyPair and IP address placeholder
# The IP address is whitelisted through the 
# LimitedSSHAccess parameter
ssh -i {KEY_PAIR_NAME}.pem ec2-user@{IP-ADDRESS}

#### Create and use demo directory
mkdir ~/demo
cd demo

#### Create dummy website
echo "<h1>Hello World</b>" > ~/demo/index.html

#### Start web server 
# The use of port 80 is defined in the
# FormationSecurityGroup 
sudo python -m SimpleHTTPServer 80

Generally we should avoid using sudo.

Screenshot of terminal that connected to EC2 instance and started a Python web server. In the background is the browser, that connects to the same EC2 instance and successfully displays the hosted website.

If everything works out as planned we are able to SSH into the virtual machine and start a web server. Connecting to the same IP address using the browser should then result in the expected website.

Awesome, we just created a CloudFormation stack! It’s time to celebrate 🥳

Delete the stack

After a celebration we need to do the boring, but necessary part — we need to clean our mess up. Fortunately it’s fairly easy, so that when we’re done testing and playing around we can just delete the stack. Removing the stack will clean up the virtual machine, including all other provisioned resources.Navigate to the CloudFormation service, select the stack and click “Delete”. This will delete all related resources. Refresh the event section after a little bit to make sure the deletion completed successfully.

Tip: Check into the billing and cost management section of AWS Console 1–2 days later and verify that no unexpected charges are accruing. Unfortunately it doesn’t seem to be possible to see real-time charge-updates.

Conclusion

There you have it, a virtual machine provisioned through CloudFormation. It configured everything from virtual machine, to the network to the IP address.

However, CloudFormation allows us to do much more. Imagine provisioning a bunch of stacks like this for a particular application, we would still have to SSH into these machines and set up a web server or run whatever application we want manually. Wouldn’t it be great if we could automate that step as well, so that we don’t even need to SSH into any of the machines? CloudFormation can help us with that as well! In a future chapter I’ll dive into how we can further customize our YAML annotation to run our scripts, after we successfully spin up a virtual machine.

Please let me know if you have questions, comments and if this short introduction to CloudFormation was helpful.

Sample Project

https://gitlab.com/skofgar/cloudformation

References

AWS CloudFormation Documentation

Others

CloudFormation 101

Thanks

Thanks to my coworker Alexandra for reviewing and giving me feedback on this article.

Originally published at Skofgar’s Blog.

Git: How to replace the master branch

Roland 🐺 — Mon, 31 Aug 2020 02:00:06 +0000

It happens to all of us, we accidentally commit something into a git repo that we didn’t mean to, even worse we also pushed it to the main repository on GitLab or GitHub.

TL;DR — too long; didn’t read

Replace the current “master” branch with another “backup” branch.

Local: git branch -f master backup

Remote: git push origin +backup:master

via: StackOverflow

Why does it matter if we commit undesired content into a repository?

Good question. We use git to keep track of our source code, facilitate collaboration and preserve how the project evolved to the current state. Having the history at our fingertips is helpful when developing new features or trying to understand a specific section in the code — granted having meaningful commit statements is important.

However there are a few things we do not want to track in git. This can entail, but is not limited to:

temporary files
the built software (binaries, executables, …)
personal information
credentials
proprietary information

Sensitive information that could accidentally or unknowingly be committed to the repository, like personal information, credentials and proprietary information may be harmful to the business should the information ever be leaked. This applies to public and private repositories — please note that even private repositories are affected, since they sometimes become public or get accessed by various people.

Can’t I just undo the change or roll back to a previous commit?

If you already pushed it to the master branch of your project, it will likely be too late. Even reverting a commit or committing an updated version where the undesired information is removed, will still leave traces in the git history.

Strategies on how to remove unwanted information

Here are a few strategies, depending how big of a correction is needed:

L1 — committing unwanted information to a local branch

Committing to a local branch is the easiest step to resolve, you can either amend the lastest commit and remove the undesired changes, or branch off from the previous commit and then delete the erroneous branch.

L2 — pushing unwanted commits to a shared repository, feature branch

As soon as commits are pushed to a shared (origin/remote) repository, things get tricky. If it is a feature branch or any other non-main (aka master) branch follow these steps:

Notify anyone that might be collaborating on this branch, that they should refrain from pushing/pulling
Follow the L1 steps and create a new branch that contains only the commits before the erroneous commit.
Push the new branch and delete the erroneous branch locally and on the shared repository
Optional: rename the new branch to have the same name as the original branch
Notify colleagues to resume work as usual

L3 — pushing unwanted commits to a shared repository, main branch

This is one gets very nerve wracking to resolve. Main branches could be any branches that are either production or development-sensitive. Often their names contain words like master, development or release, but this depends on the practices of your team.

Notify your the project lead and your coworkers that are actively working on the project to stop pulling and pushing to the main branch
If you have a Continuous Integration and Continuous Deployment (CI/CD) pipeline running, check with your colleagues if it has to be paused or active jobs have to be aborted
Follow the L1 steps and create a new branch that contains only the commits before the erroneous commit.
Replace the main “master” branch with the “fixed “branch using: git push origin +fixed:master Depending on what your remote repository is named you might need to replace origin. However remote and origin are frequent default values
Notify the project lead and coworkers to resume work and reactivate any suspended CI/CD pipelines

L4 — accidentally rewriting the history of a main branch

What if a branch gets merged into a main branch that contains a lot of undesired commits and changes? For example an experimental branch. Merging a lot of changes into a main branch may rewrite a lot of the git history and make going back to a previous commit and re-applying changes too tedious and practically impossible to fix.

Fear not! There might still be ways to salvage this. Check the following places:

Do you have a local copy or a very recent branch/fork of the main branch that could be used — even if it needed a few edits
Most backup softwares usually do hourly backups and might have an older version of the branch in question that could be restored
Coworkers might haven’t pulled the latest main branch yet or a branch that is only minimally diverged from the original main branch. Get a copy from them and re-apply — if needed — the latest commits
If nothing works you could explore using the BFG Repo-Cleaner to scrub the project from sensitive information. However, using the tool is very tedious and the history might still not be completely clean. Furthermore, it will result in creating a new repository and having everyone switch over to that one

Hopefully a backup can be found and the main branch can be restored by following the steps outlined in L3.

Humorous: The joy of pushing to master

How can we prevent accidental merges from happening in the future?

It is good to know that we can fix these mistakes, but the approaches covered earlier are tedious and nerve wracking. There must be better ways to prevent these mishaps from occurring in the first place?

Fortunately there are ways to reduce the risk of accidentally committing into the most sensitive branches. Let me suggest the following actions:

Do not push to main branches directly — This is the most obvious takeaway. Never push directly to a main branch. We’ve all done it and we’ve told others not to do it. It happens to me too and I had to fix main branches in the past
Use a branching model like Git-Flow or GitLab Flow. These approaches are detailed and even using a customized variation that works for you and your team is possible and worthwhile
Protect important branches — see GitLab protected branches and GitHub protected branches
Have merge-request reviews (or pull requests for GitHub)
Use secret detection — tools like GitLab support secret detection and security scanning

Conclusion

Protecting the main branches like master and establishing a workflow entailing review processes in order to get commits into these important branches helps prevent accidental commits including secrets, sensitive information and errors. Even with this model it is still possible, that mistakes can happen and we learned ways on how we can remedy undesired commits, how to replace entire branches and how sensitive information can be removed.

Please let all of us know in the comments below what other approaches there are, which ones worked for you and share your scary git stories with us.

References

Originally published at Skofgar’s Blog.

How to quickly replace environment variables in a file

Roland 🐺 — Sun, 23 Aug 2020 03:53:02 +0000

We know storing credentials or other sensitive values in a configuration file (e.g. Kubernetes yaml file) is bad, but how can we get values easily replaced without having to do a complicated string substitution or writing a custom Python script?

I often have personal environment variable files for projects that I use to store credentials and configurations in. Before working on a project I would the corresponding configuration file into the shell session. However, these files cannot be stored in git repositories or shared with coworkers or bots. Even worse, sometimes the repositories have files in them that need to be changed, which is dangerous, because it’s easy to accidentally commit these files.

Well as it turns out, there already is a good solution and it is called envsubst. We can use envsubst to substitute environment variable placeholders inside configuration files and we can even pipe it into other commands like Kubernetes' kubectl.

envsubst < config.txt

EnvSubst

The envsubst is part of the gettext internationalization (i18n) and localization (l10n) project for unix. It's usage is quite easy and I hope this will explain it.

Some systems have gettext with envsubst preinstalled. However, if it is missing, you can install it using a package manager. For macOS you can use homebrew:

brew install gettext

Learn more about homebrew in my article about setting about your development machine in one script

Example

Let’s say, we have an existing configuration file, that want to give to someone or use with a bot. Ideally we don’t want to include credentials in that file.

# my configuration file
server: https://gitlab.com/skofgar
username: foo_user
password: mymonkey

1. Create sample configuration file

We don’t want to check that information into a git repository nor do we want it laying around or send it to someone like this , but how can we improve this? Lets replace the information we don’t want in the file with environment variables:

server: $SERVER_URL
username: $USER_NAME
password: $USER_PASSWORD

2. Configure environment variables

Then define these environment variables either by defining them in the shell session with:

export SERVER_URL=https://gitlab.com/skofgar
export USER_NAME=foo_user
export USER_PASSWORD=mymonkey

or save them to a file (e.g. .env) and then loading them into your current shell session by using source .env

Make sure to use export, otherwise your variables are considered shell variables and might not be accessible to envsubst

Read more here: https://ostechnix.com/difference-between-defining-bash-variables-with-and-without-export/

3. Substitution

To run an actual substitution, perform the following commands.

> envsubst < config.txt  
server: https://gitlab.com/skofgar
username: foo_user
password: mymonkey

It is also possible to write your substitution to a new file:

> envsubst < config.txt > confidential_config.txt

Piping substitution into Kubernetes and other tools

It is possible to pipe the output into other commands like less or kubectl for Kubernetes (k8s).

# pipe into less
> envsubst < config.txt | less

# pipe a deployment "deploy.yml" into kubectl apply
> envsubst < deploy.yml | kubectl apply -f -

Conclusion

This is a great way to improve your Continuous Integration and Continuous Deployment (CI/CD) pipelines or just simplify your own workflow. I use this for a project where we have to replace credentials that are temporarily needed, that we didn’t want to check into the git repository and that a CI pipeline also needs to access.

Let me know in the comments what you think of this solution and if you have come across an alternative approach.

Here’s a summary of all steps combined:

# print content of configuration file
> cat config.txt
server: $SERVER_URL
username: $USER_NAME
password: $USER_PASSWORD

# load environment variables
> source .env

# replace environment variables in file content
> envsubst < config.txt
server: https://gitlab.com/skofgar
username: foo_user
password: mymonkey

# replace environment variables and write to new file
> envsubst < config.txt > confidential_config.txt

# pipe into less
> envsubst < config.txt | less

# pipe a deployment "deploy.yml" into kubectl apply
> envsubst < deploy.yml | kubectl apply -f -

Sources

Some of the sources I used for this article are:

envsubst for Kubernetes: https://serverfault.com/a/843883
gettext documentation: https://www.gnu.org/software/gettext/

Originally published at Skofgar’s Blog.

Prefix CLI output

Roland 🐺 — Mon, 22 Jun 2020 03:05:03 +0000

When interacting with the command line or writing automated scripts it’d be useful to prefix the output with the date or something equivalently useful. If you are a Linux or Terminal user you’ve probably come across the pipe symbol | , which for example allows you to apply grep filters, but it is not entirely clear how to prefix constant strings or dynamically evaluated commands like the date.

The following example shows how you’d use the pipe to filter multiple lines, to find a specific keyword

> echo -e "Lorem ipsum dolor\nfoo\nbar"
Lorem ipsum dolor
foo
bar

> echo -e "Lorem ipsum dolor\nfoo\nbar" | grep foo
foo

Prefix string

Similarly you can add text before or append text to each line using the sed text substitution command. The general structure of the command is ‘s/textToReplace/replaceWithThis/‘. However, we’re not substituting anything, instead we instruct it to prefix using the carot symbol ^.

# prefix with "Hello "
> echo World | sed 's/^/Hello /' 
Hello World

Prefix computed value

It is possible to prefix a dynamically evaluated command. To do so we need to add an evaluated string to our string replacement sed command. First, open the sed command by using single quotes, then we need to wrap the command we want to evaluate in double- and single-quotes "COMMAND" .

Single, double and back-tick quotes?

single quotes

retain literal value of each single character inside

double quotes

retain literal content except content that will be evaluated, like dollar signs or back-ticks

back ticks

an older style to trigger evaluation of it’s content. Nowadays the use of $( ) is more common

More information about quote types can be found: on this Stackoverflow post; information about backticks; the single and double quotes gnu docs

# example on how to insert a `COMMAND`
> echo World | sed 's/^/'"`COMMAND`"'/'
         literal value / | \ evaluate 
                  retain content

# equivalent $( ) command
> echo World | sed 's/^/'"$(COMMAND)"'/'

For improved readability I will use $( ) instead of back ticks.

# prefix with "Hello " and date
> echo World | sed 's/^/'"$(date)"' Hello /'
So 21 Jun 2020 22:21:39 MDT Hello World

# prefix with "Hello " and ISO 8601 date
> echo World | sed 's/^/'"$(date -u +"%Y-%m-%dT%H:%M:%SZ") "'Hello /' 
2020-06-22T04:16:04Z Hello World

Please note that the date is formatted with ISO 8601. I highly recommend that you consider formatting all dates with this standard, because you will do your future self or everyone else a favor by knowing what time formatting you used!

date -u +"%Y-%m-%dT%H:%M:%SZ"

TL;DR

To sum up, using the example above, we can use the pipe and the sed command to prefix each line with the date and “Hello “

> echo -e "Lorem ipsum dolor\nfoo\nbar" | sed 's/^/'"$(date -u +"%Y-%m-%dT%H:%M:%SZ") "'Hello /'
2020-06-22T05:00:34Z Hello Lorem ipsum dolor
2020-06-22T05:00:34Z Hello foo
2020-06-22T05:00:34Z Hello bar

Let me know if you find this useful or please share any alternative ways to prefix.

Originally published at Skofgar’s Blog.

Setting up your DEV machine in ONE script

Roland 🐺 — Fri, 03 Apr 2020 00:37:01 +0000

A new computer is great 💻🎉 — It’s a clean slate. It’s snappy. It’s great… but it is missing all the tools I rely on every day! My work requires me to jump around in a variety of tools and technology stacks, hence I need to install a lot of different tools and be able to support different development environments.

Photo by Claude Piché on Unsplash

Existing package manager for CLI tools

Over time developers load command-line tools like people are buying toilet paper these days. From maven , via tl;dr , stopping at kubectl to tmux. We use all of those little nifty tools. Hence setting up a new machine can seem scary task at times, but fear not! For CLI tools we solved that problem long long ago with Homebrew, or as you probably know it: brew

(The windows folks probably know Chocolatey)

You can install maven for example by running: brew install maven

Fear not! The same tool that installs CLI tools can install (almost) all our applications!

Brew can do more!

Homebrew uses formulae (or recipes) to install the tools you need in it’s Cellar. However, it can tap into its casks if you need Slack, Atom, Visual Studio Code, Java, 1Password or SourceTree — enough with the puns! It’s literally as easy as installing a CLI package.

You probably heard about casks or seen it fly by in the console.. let’s look at an example. Let’s say you want to install Firefox using brew. All you need to type is:

brew cask install firefox

Brew will update its formulae, look for the application you want to install and do it right away.

> brew cask install firefox
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> Updated Formulae
abcmidi cataclysm contentful-cli llvm logtalk
==> Updated Casks
google-chrome           

==> Downloading https://download-installer.cdn.mozilla.net/pub/firefox/releases/
######################################################################## 100.0%
==> Verifying SHA-256 checksum for Cask 'firefox'.
==> Installing Cask firefox
==> Moving App 'Firefox.app' to '/Applications/Firefox.app'.
🍺 firefox was successfully installed!

Brew can’t install my application!

> brew cask install xcode                         
Updating Homebrew...
Error: Cask 'xcode' is unavailable: No Cask with this name exists. Did you mean “swiftformat-for-xcode”?

Unfortunately not every app has a recipe on homebrew. However, you may just don’t know the correct name or have spelt it incorrectly. To double-check there is a useful brew search THE-THING-I'M-LOOKING-FOR and as long as it's not cat-pictures, you'll likely be successful.

In docker’s case, it will even give you formulae’s and casks:

> brew search docker                                                  
==> Formulae
docker docker-machine-driver-vmware
docker-clean docker-machine-driver-vultr
docker-completion docker-machine-driver-xhyve
docker-compose docker-machine-nfs
docker-compose-completion docker-machine-parallels
docker-credential-helper docker-slim
docker-credential-helper-ecr docker-squash
docker-gen docker-swarm
docker-ls docker2aci
docker-machine dockerize
docker-machine-completion lazydocker
docker-machine-driver-hyperkit
==> Casks
docker homebrew/cask-versions/docker-edge

Ok, we get it, but where is that single script?

Ok… I’ve digressing about brew. To create a single setup script, it also needs to be able to install homebrew itself. Fortunately, that’s easy and available directly from brew’s website https://brew.sh

I combined everything I mentioned in a sample script, that I use myself:

An example script for brew using formulae and casks

This drastically reduced the amount of time to set up my environment (about 25 min, depending on internet connection) and will make it easier in the future, if I decide to rebuild the machine.

Potential improvements

Here are a few improvement ideas:

Identifying all the casks that require password and moving them at the beginning would simplify this script
Further improve the script to do all additional customization (desktop, os settings, …)
If you always work on the same projects, creating and executing a script with git clones and creating all the environments might be a good next step

I hope you find this article interesting and please let me know if you have any questions.

Resources

Formulaes Overview: https://formulae.brew.sh
Formulaes for CLI Packages: https://formulae.brew.sh/formula/
Casks: https://formulae.brew.sh/cask/

Originally published at Skofgar’s Blog.