DEV Community: Yehuda Neufeld

Setting up a fresh server for Continuous Deployment with Git

Yehuda Neufeld — Sun, 05 Jul 2020 08:32:08 +0000

We all want to be able to focus on writing code, without wasting time when deploying changes to production.

But it's not that simple. There are several operations our code needs to go through going from development to production (I'm referring to a Laravel Application, but it will be similar for most web applications):

Push our new code to the server.
Update our Javascript & PHP dependencies.
Compile our Javascript and CSS.
Finally, we need to migrate the production database.

Doing all these operations manually before every deployment quickly gets too time consuming. Besides, it's a repetitive task that should be automated...

So for quite a while I researched & tried to find the best way to do this. At first my solution was using GitHub webhooks, but that solution had it's problems.

Last week, I finally found the perfect solution, which lets me deploy the code to production with just 2 keystrokes: gp 😎.

How it works

We're going to use Git (version control software). Git has several hooks that it can call after different stages automatically. We'll use the post-receive hook which is called after your repository has received pushed code.

A Hook is a program you can place in a hooks directory to trigger actions at certain points in git’s execution.

In this script we'll do all the above operations.

1. Login to your remote server

Open your terminal and login to your server using the following command:

ssh <your_user>@<server_ip_address>

Make sure that you have Git installed on the server.

2. Create a folder for your code

The source code for your application needs to be put somewhere. By convention, code goes inside the /var/www directory. Navigate there using:

cd /var/www

Now, create a new folder to put the source code. For this tutorial, it will be called app-folder.

mkdir app-folder

Now let's remember the full path to our app-folder is /var/www/app-folder/.

3. Initialize a git repository in your home folder

We'll create a git repository in our users' home folder. Let's create a repo folder to hold all our future repos:

mkdir -p ~/repo/my-app.git

Now, we'll navigate into our repo folder, and initialize a bare git repo:

cd ~/repo/my-app.git
git init --bare

Congratulations, we now have the git repo ready for use!

4. Create hook

After initializing the git repository, new folders should appear inside ~/repo/my-app.git/. Navigate to the hooks directory, and create a new file called post-receive using your preferred text editor.

cd hooks
vim post-receive

Make sure to spell post-receive right, because I misspelled it - post-recieve, and it took me a while to figure out why it wasn't working 😃...

Now for the serious part: we're going to write the program to deploy our application.

This is what my post-receive script ended up looking like (as mentioned above - this is for a Laravel app):

#!/bin/bash

TARGET="/var/www/app-folder"
GIT_DIR="/home/<username>/repo/my-app.git"
BRANCH="master"

while read oldrev newrev ref
do
        # only checking out the master (or whatever branch you would like to deploy)
        if [ "$ref" = "refs/heads/$BRANCH" ];
        then

                cd $TARGET
                pwd

                echo "activating maintenance mode"
                php artisan down

                echo "Ref $ref received. Deploying ${BRANCH} branch to production..."
                git --work-tree=$TARGET --git-dir=$GIT_DIR checkout -f $BRANCH

                echo "Running NPM build command"
                npm install
                npm run build

                echo "running composer install"
                /usr/local/bin/composer.phar install --no-interaction --no-dev --prefer-dist

                echo "running migrations"
                php artisan migrate --force

                echo "going back live..."
                php artisan up

        else
                echo "Ref $ref received. Doing nothing: only the ${BRANCH} branch may be deployed on this server."
        fi
done

The code is pretty self explanatory, as it is echoing out what it's doing.

Save the file, and make it executable by running this command:

chmod +x post-receive

The work on your server is done.

5. Push local code to the server

Back on our local machine, navigate to your project folder, and initialize a git repo (if it isn't already). Then we need to add our remote server as a remote:

git remote add <name-of-remote> ssh://<username>@<server_ip_address>/<path_to_git_directory>

# For our example:
git remote add azure ssh://<username>@<server_ip_address>/home/<username>/repo/my-app.git

I called my remote azure, as it's an Azure server.

Now, whenever you want to deploy your code, just run:

git push <name-of-remote> <name-of-branch>

# we can omit the branch name - it will default to master
git push azure

One second, this is more than 2 keystrokes!

Oh, you're right... to achieve that make your remote the default upstream, by running:

git push --set-upstream azure

Then you can deploy to production by running:

gp

It works, because gp is an alias to git push (if it isn't, you can add this alias yourself).

If you don't want to set the remote-server as the default upstream (you may want GitHub to be the default), then you can leave the default upstream to be origin (GitHub) and deploy to your server by running gp azure.

You even get all the output right into your local terminal! Here is the deployment in progress:

Summary

I'm writing this for my future reference, as this is so easy to setup, and makes deploying to production a breeze.

Sidenote

If you get errors relating to permissions, and to avoid having to use sudo in the web directory on your server, here is a solution with security in mind:

Add yourself to the www-data user group, and set the setgid bit on the /var/www directory such that all newly created files inherit this group as well.

# add yourself to the www-data group
sudo gpasswd -a "$USER" www-data

# apply the ownership to the /var/www folder
sudo chown -R "$USER":www-data /var/www

# Correct previously created files (assuming you to be the only user of /var/www):
find /var/www -type f -exec chmod 0660 {} \;
sudo find /var/www -type d -exec chmod 2770 {} \;

source

Wrapping my Brain around Recursion

Yehuda Neufeld — Fri, 05 Jun 2020 08:39:48 +0000

P.S. This is my first post published on dev.to. If you enjoy this article, check out my personal blog for more stuff! 😃

When I was at the start of my programming career, whenever I saw something about recursion, I would run away as fast as I could — 😲 RECURSION 😲 — what could be scarier than this complicated concept?

One day (about 2 years into my programming journey), I was coding along, when suddenly an unexpected thought popped up in my brain: wait! recursion is the perfect way to solve this problem/challenge!

The Challenge

I was building a desktop client to edit wikipedia pages (or any wiki powered site) offline. The way it works is like so: I present the user with a list of categories that exist on the site. The list is a tree, where every category can have subcategories, and subcategories have subcategories, and so forth. The category tree is many levels deep, and every branch ends with a list of pages in the category.

The user would then select some categories, and press a button to save them all to his computer for offline use.

So the first thing we need to do — is to build up the category tree & store it in a json file for the program to use.

I found a Node.js Wiki API library called nodemw, which exposes lots of nice methods to interact with the API.

We can use this method the library provides us:

getSubcategories: this method takes the category title as a parameter, and returns an array of all subcategories & pages that are members of this category. Unfortunately, the wiki API doesn't have an endpoint for retreiving all nested categories at once, we can only get 1 level of children at once. But we need to get all the subcategories however deep they might go...

The Brainstorm

This is a complex challenge, so let's try to break it down into small parts:

Get subcategories of the root category.
For each item in the returned array: Each item could be either a subcategory, or a page. So we need to check:
- If it is a page, do nothing.
- But if it's a subcategory — then we need to get its subcategories — in other words: go back to step 1! 💡💡💡 (when you arrive here the 2nd time — go back to 1 again, forever 😲)
- But that's not all: after we get its subcategories, we need to assign it to its parent category, because we're building a tree, right? Here is where it clicked in my head: This is recursion!!!

We need to repeat a task over & over again for an unknown amount of times, until a certain condition is met. If we don't put a solid condition in place - the computer might repeat over the task infinitely — until the computer crashes, or until there is a stack-overflow 🤯 (between ~10,000 - ~30,000 levels).

Implementation

Once we understand the concept - we can get to write the code to do the task 😎. And oh is this part complicated... especially when we need to fetch data over the network in each iteration of the loop...

I spent 2 days overheating my brain writing this function... There were countless sub-challenges that I had to break through in the process. But first let's have a look at the final working code 👀:

I added comments in the code, instead of explaining every line separately.

async rebuildCategoryTree() {

  // 1. Get children of root category
  const rootCats = await this.$wiki.getSubcategories('עץ קטגוריות ראשי');

  // 2. Here we define the recursive function
  const getChildren = async cats => {
    // do some important checks 
    if (cats.some(cat => cat.type == 'subcat' && !cat.subcats)) {
      console.log('diving deeper...', cats);
      for (const cat of cats) {
          if (cat.type == 'subcat') {
            let subcats = [];
            let cachedCat = this.cachedCats[cat.title];
            if (cachedCat) {
              // we already fetched these from the server
              if (cachedCat.visited) {
                // STOP! we have an infinite loop!
                console.log('TERMINATING INFINITE LOOP', cat.title);
                return cachedCat.subcats;
              }
              console.log('...Using cache', cat.title);
              // my solution to overcome infinite loops (circular relations)
              cachedCat.visited = true;
              subcats = cachedCat.subcats; // use from cache
            } else {
              console.log('...Fetching fresh', cat.title);
              // THE ACTUAL WORK - fetch category members
              subcats = await this.$wiki.getSubcategories(cat.title);
              // add to cache
              this.cachedCats[cat.title] = {
                ...cat,
                subcats,
              };
            }
            // calls this function itself - to recursively fetch children
            cat.subcats = await getChildren(subcats); 
          }
        }
    } else {
      console.log('coming out...', cats);
    }
    return cats;
  };

    // 3. Start working!
    // this tiggers the recursion
    const catTree = await getChildren(rootCats);

    // 4. Save the resulting tree as a json file
    this.writeFile(catTree);
}

Sub-Challenges

Here are some of the challenges that arose while writing the recursive function:

1. Circular Relations (infinite loops):

I was testing the code, and it seemed to be working, but it was taking forever... Although the wiki site I was working with has 83,691 pages, it still seemed to be taking too long.

After debugging a while — I caught the culprit. Not a bug in my code, but on the wiki site. They had a few circular relationships (where category A had category B as a child, which had category A as a child...) which caused an infinite loop 🤯.

At first they fixed it on the wiki site, but I still couldn't guarantee that such loops don't exist in other places in the tree. So I had to come up with a solution to detect such loops, and get out of them in time...

The solution presented itself from a different thing I wanted to fix: not to lose all progress if the function stops unexpectedly (by a network error etc.). For that I started saving every getSubcategories result which came back from the server to an object I called cachedCats (which caches the results from the network, so if it gets interrupted & we need to start again - we shouldn't need to start from scratch).

Once I had that, I could use that object to know if we already visited that branch in the tree. Albeit not the first time, because we might be after an unexpected error, so the 1st time we encounter something in the cache we record that visit (by adding visited: true), then if we encounter the cache with visited == true we know that we've caught an infinite loop! So we log a scary message and return early:

if (cachedCat.visited) {
  // STOP! we have an infinite loop!
  console.log('TERMINATING INFINITE LOOP', cat.title);
  return cachedCat.subcats;
}

2. Asynchronous Operations:

As we're fetching data from a remote server - which take time - there's more complexity.

The nodemw library was written quite a few years ago, so all methods were written in the old callback style, where each method takes a callback function as a parameter, which gets called after the data comes back from the server. The callback function in turn gets 2 parameters: error, data, so we first need to check for the error, only if not present we can assume that we got the data.

// BEFORE:
// callback based function
getChildren((err, cats) => {
   // first handle the err (if it exists)
   if (err) {
     console.error(err);
     return;
   }
    // Now we can use the categories...
    cat.subcats = cats
});

This was nearly impossible to use in a recursive function, because every callback will get called separately at a different time... (probably possible, but makes it much more complicated).

At first I thought I'll fork the nodemw repo, and update it to use Promise based async functions. I actually forked the repo, but the work to rewrite the whole library in Promises is easier said than done...

Luckily, I discovered that Node's utils module has a method to do exactly that! 😃 Just pass an old style callback function to utils.promisify, and you get back a modern Promise based function!

So I wrote a thin wrapper class around the nodemw library that converts the few methods I needed to Promise based functions:

class wikiAPI {

    /**
   * Login the user to the wiki
   * @param {string} username
   * @param {string} password
   */
  logIn = util.promisify(this.Bot.logIn).bind(this.Bot)

  /**
   * Edit a page on the wiki
   * @param {string} title - Page title.
   * @param {string} content - Page content
   * @param {string} summary - Summary message
   * @param {boolean} minor - If it's a minor edit
   */
  edit = util.promisify(this.Bot.edit).bind(this.Bot)

    //...

}

(I added jsdoc documentation, to enhance vsCode's intellisense)

Now I could use async await , which made things mush easier & cleaner. 😀

// AFTER:
// Aaaaahhhhh.... much better!
cat.subcats = await getChildren(subcats);

Summary

It was really challenging to get everything working perfectly, but it's so worth it to tackle these types of challenges head on — you learn so much & get a deeper understanding of Javascript concepts.

In the end, it's amazing to watch your function do its job, recursively building a massive tree, doing a task thousands of times, deciding by itself when to go deeper, and when to come back up a level...

That's the power code!