DEV Community: Shavant Thomas

Getting Started in CI/CD using CircleCI

Shavant Thomas — Mon, 29 Jun 2020 00:15:47 +0000

CI/CD = CI/CD^2 = CI/CD/CD???

Exactly what is CI/CD? Sure many know what the short version stands for and even then it can be a little confusing. For instance maybe you get Continuous Integration. It allows small and/or quick changes to be committed and integrated into your main branch . There is some confidence in the integrity of your code; because it should be constantly going through unit and integration testing. It is also able to build consistently. What about CD however? Possibly you have heard two different definitions. However they can sound very similar if not the same just from the naming alone. So is it Continuous Delivery or Continuous Deployment? Well it's both and while its often used interchangeably they are different. In tech many things are kept short and/or abstract. So while everyone uses the short version to keep it concise, I like to think about it really as CI/CD/CD or CI/CD^2. Translation Continuous Integration, Continuous Delivery, Continuous Deployment.

Continuous Delivery

Continuous Delivery builds on CI. CI gives a nice automated process which produces a sanity build of your software. Now you need a nice auto delivery of the product. Let us refer to Continuous Delivery as CD for this part. With a small team its easy to imagine various states going on all at once. Someone can be checking in code while another has some test going on etc etc. We need a way to see the updates and possibly how it works in a central live environment. For many this is called the staging environment. The staging environment is usually where anyone on the team can test and review before we deploy to production for all to see. However we don't want someone constantly waiting to act on this; nor do we want everyone's hands in the pot do we? Instead we want a predictable routine. This includes some defined rules which can be viewed by anyone such as a schedule for instance. Using reproducible steps which run the same on each iteration. We want this process to be predictable resulting in an expected quality product each time. Managing environment, services, etc, should also be included in CD. By having this automated delivery process we can have a happier and more productive team. Just how work is constantly being produced and changes are happing at any moment. The pipeline is set to constantly keep churning out new versions of the product.

Continuous Deployment

Many stop at Continuous Delivery and with good reason. Not everyone is ready or even desires Continuous Deployment. Where as Continuous Integration and Delivery were about how the organization can benefit and improve workflow productivity. Deployment now involves the users. Continuous Deployment is the next step up and is about deploying to your production environment. With no one to interfere the product quickly goes from being in house to now being live for all users. As you can imagine their is a huge reliance on automation for all this to happen. Therefore due diligence needs to be done by reviewing the entire pipeline. To each their own but some may not want or can't implement this due to standards and regulations. Others may not have the automation in place for infrastructure or rollbacks in case something fails. One can name a ton of possibilities as to why or why not to implement. Whatever the reason Continuous Deployment rewards should be weighed heavily along with the risks.

Why CircleCI?

I have dealt with my share of CI/CD tools but for this post CircleCI will be the focal point. It's the one I have become accustomed with the most lately. While I am sure down the line I will experiment with another tool again I must admit I really like it. It has a ton of features I like but something I really appreciate is just how easy it is to get started with a new project. In this post I'll walk through how to get up and running locally as well as with the cloud platform.

Why a local setup? There are a few benefits but speed and convenience are big ones. Having some minimal quick access to debug files and config changes, without having to commit and push each time were reason enough to pique my interest. The time investment to set up locally is also very little in the grand scheme of things; So why not get the most out of it? Of course you will need to sign up for a CircleCI account (circleci.com) which is free to start building. It can also instantly be used with a Github or bitbucket account.

Install CircleCI with HomeBrew

There are several ways to install CircleCI locally but I did it through good old trusty homebrew. Because I had Docker for Mac already installed. It's actually recommended that you use the following command.

brew install --ignore-dependencies circleci

If you don't have Docker for Mac install just do

brew install circleci

Now lets validate that you indeed have it installed and that everything is working as expected.

You are going to want to obtain your Personal API token. Once signed into your CircleCI account, navigate to User Settings → Personal API Token and generate a new one, if you don't have one already. Make sure to copy it somewhere, since you won't be able to see it again. If lost you will need to regenerate a new token.

Personal API Tokens.

With the newly acquired Personal API Token you can now configure circleci. Run the command below and it will start a quick walkthrough. If you are not setting this up on a dev server or in the cloud yourself, the default circleci website supplied is your host.

circleci setup

Validate that your local setup works. In a project root directory create .circleci/config.yml. For validation purposes just copy & paste the barebones setup below which you will want to change later to suit your purposes.

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/node:12.16.1 # the primary container, where your job's commands are run
    steps:
      - checkout # check out the code in the project directory
      - run: echo "hello world" # run the `echo` command

Running the validate command in the root of your project directory should find the config file and return a statement that the file is valid.

circleci config validate
Config file at .circleci/config.yml is valid.

Conclusion

You now have a working CI pipeline locally. However thats not the only thing you accomplished. Remember earlier I stated I liked how easy CircleCI makes it to get up and running? While this post guided you through a local setup. You already have what you need to start using the cloud hosted platform as well. With your Github or Bitbucket account you used to sign up for CircleCI. Once you push the branch up to your repository with the changes you made in ".circleci/config.yml". Your cloud account will also do its thing and run the pipeline. Pretty much you are already up and running. Go ahead and give it go. Moving forward as the code gets updated, you should also make the proper changes to update the config.yml as needed. This is just the start and will be a series of post. As we progress, we will walk through some of those changes together. Please feel free and let me know any thoughts, comments, questions. Till next time!

Terminal++ with iTerm2 & ohmyzsh

Shavant Thomas — Sat, 04 Apr 2020 20:01:27 +0000

One thing I love about getting a new dev machine is starting off fresh. After getting my new MacBook Pro one of the first things I look forward to is setting up my dev environment. For me this starts with my shell & terminal setup.

PreReqs
iTerm2
OH MY ZSH!
What's your Theme?
- Choose a Theme
- Install your Theme
Time to Power Up!
- Powerlevel10K
The Wrap Up

PreReqs?

If you don't have Homebrew installed you will want to do that first as the rest of this post assumes so. Homebrew pauses and explains what its doing so expect interaction on your part as well.

Copy & Paste the snippet below.

xcode-select --install
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Upgrade your Terminal - iTerm2

By this I mean if you're still only using the basic terminal that comes with your Macbook its time to consider making the switch to iTerm2. Before you ask why just take a visit to the features page on the website.

Download and Install using this link: https://www.iterm2.com/downloads.html

Use cask from brew to install.

brew cask install iterm2

OH MY ZSH!

To complete the base package this gem should be in your toolkit. Firstly make sure that you already are using a zsh shell. For new macs this should already be your default. If you're upgrading to Catalina or have an older OS you may need to switch your shell. In your new downloaded iTerm2 shell simply confirm with an echo command.

echo $SHELL
/bin/zsh

You are good to go as long as "/bin/zsh" was returned. If another shell was returned you either need to change your default shell

chsh -s /bin/zsh

or install zsh before proceeding.

brew install zsh

Now its time to proceed with what we are really here for; oh-my-zsh. If iterm is like getting the combo meal. Then oh-my-zsh is like ordering from the secret menu. You can take a look at their page (https://ohmyz.sh/) and/or just google to see what the fuss is all about.

Install oh-my-zsh:

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

You should notice the change to your shell now but if not just restart iterm2. Now you got your tools and it's time for the fun part. Customization!

Note: The Rest of the article follows my personal setup. Follow along if you are interested in a similar configuration or tips on how to customize your own setup.

What's your Theme?

Choose a Theme

Let's start off by choosing a theme. Themes can be found at https://iterm2colorschemes.com/. To install simply navigate by going to:

iTerm2 → Preferences → Profiles → Colors → Color Presets → Visit Online Gallery.

This will take you to the same page mentioned above. Near the top of the page you will see some folders named "tar.gz" and ".zip". By clicking on either of the folders you are able to download the entire bundle of themes on the page. Of course if you are already on that page you can skip all of that navigation as well.

Install your Theme

iTerm2 → Preferences → Profiles → Colors → Color Presets → Import → →schemes →

After this when you go back to Color Presets you will now see the theme you want as an option to be selected.

Time to Power Up!

oh-my-zsh has its own set of themes that can give a boost to the look and feel of your terminal. The default along with many others can be found on the git documentation page but there are more out there if you are willing to search.

GitHub Documentation:

https://github.com/ohmyzsh/ohmyzsh/wiki/Themes

Powerlevel10K

My theme of choice is powerlevel10k. To install clone the repo

git clone --depth=1 https://github.com/romkatv/powerlevel10k.git $ZSH_CUSTOM/themes/powerlevel10k

Next in your ~/.zshrc set the theme

ZSH_THEME="powerlevel10k/powerlevel10k"

Save and source

source ~/.zshrc

Upon success powerlevel10k will run a wizard for some setup configuration and will ask to install the Meslo Nerd Font; which is the recommended font for this theme.

Example Screenshot of How my Terminal Setup looks

The Wrap Up

Pretty much everything you need for an awesome terminal is now at your disposal. Of course there are far more customizations that can be done to tweak your setup and this will set you well on your way. Please share any comments or questions. I also plan on sharing more similar to this and possibly making a series for these kind of post. Let me know if there is something in particular that piques your interest.

Be Aware of these MongoDB Gotchas

Shavant Thomas — Sat, 01 Feb 2020 08:00:20 +0000

What is this about?

We all have had those times where we take on a task and at first glance it seems like this should be straight forward. Suddenly you are left with a million questions and many internet searches you've entertained have led you down a wormhole with no solutions in sight. In my case this came about from a MongoDB migration in which I had to lead.

Prior to this we hosted MongoDB on a few instances with collections for all our maintained environments. One of the most important projects for the year meant we would be expecting a large influx of users which we were not ready for. The need for higher availability, scaling and speed were critical.

Enter MongoDB Atlas. Before proceeding any further let me start off by saying that MongoDB has some pretty good documentation; So I will be skipping over most of the "Getting Started" which can be found in MongoDB docs. While I won't discuss every single obstacle. Hopefully I can save others some precious time where I had little to spare.

TLDR version:

A Few Gotchas to watch out for when doing a migration or starting out with MongoDB Atlas

Using a VPN? - Always check the Whitelist!

I figured let's start off with my homer Simpson moment. VPN is used to access any of our instances and I am pretty much always running one. At first it took me a while as I worked through basic connections steps. However I soon realized this was my classic "D'OH" moment and I needed to whitelist my IP and make a mental note that when I see a connection error. Update it!

Fortunately MongoDB makes it really easy to do this. Just a few clicks and you can add your current IP Address.

Check and match all your versions. It's a Must!

(If you're already using containers such as Docker you can skip this part about Matrix Hell. We were not!)

Remember those match card memory games? In the game you needed to match up every pair and complete the whole set to win. I found the same to be true in my case of dealing with MongoDB. Through a ton of trial, documentation and yes even contacting support(which basically they just reiterated what I already had learned and attempted through documentation). It came down to me tracking down versions. Ruby, mongodb, mongoid, linux distro, mongo shell.... Just basically check everything. As I was also in the process of upgrading our mongo version; this meant format changes had to be applied to the mongoid yaml configuration. An example template of this can be found at this link.
Mongoid.yml

Connection issues? - Try working your way through a manual connection.

While MongoDb has some pretty good documentation there is still room for improvement. The suggestions for how to connect your app just were not working for me. Even support pretty much just repeated what I already had attempted thus far but they also never asked about what version I am using to help connect the dots.

Finding the difference between when to use "mongodb+srv://" or using "mongodb://" is a pretty significant piece of info. It was only when comparing my working manual connection with the sites suggestions for setting up a config that I noticed this key difference. Replica sets, required info and options are all dealt with differently depending on which one you use. I feel this has some key differences that maybe could have been highlighted in a stronger way. As it took the efforts of using a fine tooth comb to help figure out the discrepancies.

Just take for example a barebones connection string.

Older version:

mongodb://mongodb0.example.com:27017,[mongodb1.example.com:27017](http://mongodb1.example.com:27017/),[mongodb2.example.com:27017/admin?replicaSet=myRepl](http://mongodb2.example.com:27017/admin?replicaSet=myRepl)

Newer version:

mongodb+srv://mongodb0.example.com/[admin](http://mongodb2.example.com:27017/admin?replicaSet=myRepl)

With the new version you only really need the address but the older one has you explicitly define each replica node.

The best thing to actually do was just break out my manual connection command in its entirety and put it in the config. Had I never looked at their suggestion I probably would have saved myself a bit of a headache.

MongoDB has Connection Instructions which lets you choose your desired method.

Remember your old services

Don't forget during a migration your sometimes launching the new while still working with the old. When you run into some funky behavior. Check to see if you still have the older version up. For example if you use Redis with MongoDB. Which ever way you chose to implement could possibly mean sharing the service between old and new instances. This will have your app and/or data performing chaotic at times.

Use Drop to Update

If you are doing several imports/restores over the course of the migration. You may not want to delete your collection every single time. I have a preference for full delete & restore when finalizing my migrations. However if I just want to test/verify the most recent data; an update is all that is desired. Problem is MongoDB does not have an update in the traditional sense. During restore MongoDB does not allow overwriting of records. Try to run a restore and you will encounter a number of duplicate key errors.

An Example error:

- E11000 duplicate key error collection: mydb.mycollection index: id dup key: { : "MyKey" }

To avoid the annoying errors you just need to add the —drop flag. This removes the collection from your target right before it restores from your source dump.

Ex.

mongorestore --host <replSetName>/<hostname1><:port>,<hostname2><:port>, authSource=admin" —drop <dbname> <dbpath>

Bonus

This is not really a gotcha at all. It just feels weird to show the restore command without the dump especially since you have to dump before you can restore. By default mongodb provides a dump directory with the contents for you if a path is not provided.

mongodump --db <dbname>

Good Luck!

Well time to wrap it up! Hopefully this helps someone out there even if its just to confirm nope you are not crazy. Please let me know if you have any questions or if something is not clear in the post.