Yankee Maharjan

Posted on Dec 26, 2020 • Edited on May 11, 2021

Streamline your projects using Makefile

#productivity #programming #tutorial #cli

make is one of the tools that we use heavily for streamlining tasks on our projects. It has proven to be helpful specifically for streamlining the development process, repeating mundane tasks with custom CLI like subcommands and mainly onboarding new team members.

With a set of rules in Makefile, you can get up and running in no time, keeping the process sane and saving time and effort for everyone in the team. We'll be going through the basics to some interesting stuffs we can do with Makefile.

There are two pieces to this equation, one is the make CLI tool and the next is the Makefile . The basics is make reads the rules from the Makefile and executes them. What I will be showing today is just a small part of what make is capable of.

Writing Makefile

If you have worked with YAML files before then you will feel right at home writing Makefiles.

Anatomy of Rules

Every Makefile consists of rules with the anatomy of:

target: dependencies
    recipe

target: target can be an executable, object or just a name for an action that we want to carry out. We will be using targets purely with the placeholder name for the rule. Be mindful about the name, as it should resonate with the action we want to perform with no confusion whatsoever.
dependencies: Dependencies are the rules that needs to be executed, in order for the current rule to work.
recipe: recipe is the meat of the Makefile, it is the action that we want to perform with our target name. Make sure to put a tab character at the start of every recipe line (just like YAML). You can also replace the tab character with anything you want using the .RECIPEPREFIX variable.

Next we will be looking into some examples on how to make use of Makefile. These examples will be based on setting up development environments.

Basic Rules

A basic rule where you just want to put some alias is straight forward.

Let’s say you have a python project and you want to hand it over to a new team member. How do you streamline the setup process. Maybe it can look something like this.

Note:
# is for comments.

@ symbol is to disable printing the recipe to stdout.
Test without the @ symbol at the beginning of the recipe.

:= is the expansion operator which prevents using subsequent value with the same variable name.

SHELL variable determines the default shell to execute the recipe.

SHELL :=/bin/bash

.PHONY: format check

venv: # setup a virtual environment
    @python3 -m venv venv

setup: # install dev dependencies
    @pip install -e .[dev]
    @echo -e "\nInstalling pre-commit hook..."
    @pre-commit install

format: # format code using black
    @black .

check: # check for formatting using black
    @black --check --diff -v .

test: # run pytest
    @pytest -vvv

You can do something similar with your existing project.

Now to get up and running, all you have to do is:

$ make venv 

$ . venv/bin/activate 

$ make setup

$ make format 

# and so on

Rules with Dependencies

Taking the reference from the example above, suppose we want to print out the output of check target every time we run the format target. So how do we create that dependency? It’s plain simple, we just have to update the format target to look something like this:

format: check # run the formatter on files.
 @black .

We have added the dependency of check to the right of the target, just like showcased on the anatomy Anatomy of Rules section.

Variables

We can also define variables if we have some piece of command for repeated use. For this example we will be taking the reference of the Django management command.

Variables are normally written with all caps and uses := to assign variable name to a value. Variables can be accessed using either $() or ${} syntax.

DJANGO_MANAGE := python manage.py
run: 
 @${DJANGO_MANAGE} runserver

show: 
 @${DJANGO_MANAGE} showmigrations

migrate: 
 @${DJANGO_MANAGE} migrate

Also your SHELL environment variables are converted in to Makefile environment variables, so you can directly make use of them while creating your rules.

Example:

In our shell we can export an environment variable called INFO.

$ export INFO="Run make help to show all the available rules."

And now in the Makefile we can refer to it as any variable.

info: # show project info
    @echo ${INFO}

Default target

If you just run make on your command line nothing is going to happen. But we can change that by using the .DEFAULTGOAL special variable and assigning the target we want to run by default.

.DEFAULT_GOAL := run

Now, next time you run make it is going to run the Django server by default.

Self documenting

Now we have bunch of targets on our Makefile and we also called this combo as a custom mini CLI app. Wouldn’t it be great, if we could have a help command similar to a real CLI app? Say no more, thanks to the blog from Victoria Drake we have the script to do so.

Just create a help target and assign it as a .DEFAULT_GOAL. With this, all the comments we have been writing on our target gets converted into a nice help message.

.DEFAULT_GOAL := help 
help: # Show this help
 @egrep -h '\s#\s' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?# "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'

Include other Makefiles

We can separate out Makefiles based on the tasks they perform and include them into the main Makefile. We usually have separate Makefile managed for environment variables, Docker and Kubernetes. This offloads all the tasks from project set up to Deployment to the Makefile.

I will show a brief example of each of the file just to give an example:

Note: Since make runs each recipe on a new instance of the shell, we can lazy evaluate the variables using ?= meaning, they are initialized only when referenced for a single shell instance.

Makefile
Root makefile composed of other Makefiles.

SHELL :=/bin/bash
APP_ROOT := $(PWD)
TMP_PATH := $(APP_ROOT)/.tmp
VENV_PATH := $(APP_ROOT)/.venv

export ENVIRONMENT_OVERRIDE_PATH ?= $(APP_ROOT)/env/Makefile.override

-include $(ENVIRONMENT_OVERRIDE_PATH)
include $(APP_ROOT)/targets/Makefile.docker
include $(APP_ROOT)/targets/Makefile.k8s

Environment Variables
Makefile.override
Makefile containing just the essential environment variables.

STAGE ?= <stage>
SERVICE_NAME ?= <service-name>
AKS_RESOURCE_GROUP ?= <resource-group>
AKS_CLUSTER_NAME ?= <cluster-name>
REGISTRY_URL ?= <registry-url>
AZ_ACR_REPO_NAME ?= <repo-name>

Docker
Makefile.docker
Makefile containing docker rules.

export GIT_COMMIT ?= $(shell cut -c-8 <<< `git rev-parse HEAD`)
export BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)

export DOCKER_BUILD_FLAGS ?= --no-cache
export DOCKER_BUILD_PATH ?= $(APP_ROOT)
export DOCKER_FILE ?= $(APP_ROOT)/Dockerfile

export TARGET_IMAGE ?= $(REGISTRY_URL)/$(AZ_ACR_REPO_NAME)/$(SERVICE_NAME)
export TARGET_IMAGE_LATEST ?= $(TARGET_IMAGE):$(BRANCH)-$(GIT_COMMIT)

acr-docker-login:
    az acr login --name $(AZ_ACR_REPO_NAME)

docker-build:
    docker build $(DOCKER_BUILD_FLAGS) -t $(SERVICE_NAME) -f $(DOCKER_FILE) $(DOCKER_BUILD_PATH)

docker-tag:
    docker tag $(SERVICE_NAME) $(TARGET_IMAGE_LATEST)

docker-push: acr-docker-login
    docker push $(TARGET_IMAGE_LATEST)

Kubernetes

Makefile.k8s
Makefile containing rules for Kubernetes.

export OVERLAY_PATH ?= $(APP_ROOT)/k8s/overlays/$(STAGE)/

define kustomize-image-edit
    cd $(OVERLAY_PATH) && kustomize edit set image api=$(1) && \
    cd $(APP_ROOT)
endef

kubectl-apply:
    kustomize build $(OVERLAY_PATH)
    kustomize build $(OVERLAY_PATH) | kubectl apply -f -

update-kubeconfig:
    az aks get-credentials --resource-group $(AKS_RESOURCE_GROUP) --name $(AKS_CLUSTER_NAME)

aks-deploy: update-kubeconfig
    $(call kustomize-image-edit,$(TARGET_IMAGE_LATEST))
    make kubectl-apply

aks-delete: update-kubeconfig
    kubectl delete namespace $(STAGE)-api

kustomize-edit:
    $(call kustomize-image-edit,$(TARGET_IMAGE_LATEST))

Now we have orchestrated all these Makefiles, it is easier to keep track of all the rules and makes working with Makefiles sane, if you are doing a lot with it.

Conclusion

So with the use of Makefile we can streamline a lot of redundant tasks in our projects without having to remember overwhelmingly long and varying commands.

It increases the productivity of the whole team; with easier project setup and redundant tasks outsourced to the Makefile with intuitive target names, leaving the devs to focus on more serious tasks at hand.

Top comments (13)

Comment hidden by post author - thread only accessible via permalink

Mark Smith • Dec 27 '20

I like the general idea of where this tutorial was going but there were some parts that either were not well described (@ symbol) or inconsistencies (env variables) that broke the flow, making it difficult to understand.

Yankee Maharjan • Dec 27 '20 • Edited

Hey Mark, thank you for the comment, I have written about `@ symbol on the Notes section here.

For environment variables, from where did you find it confusing? Is it the sub topic Variables itself or Include other Makefiles?

Let me know, I will update accordingly to make it more understandable. 🙂

Mark Smith • Dec 27 '20

@ symbol is for disabling the echo for recipe.

I have no idea what that means. It feels weird adding symbols to bash commands like that, where is the symbol interpreted?

In our shell we can export an environment variable called greeting.

But then the name of the variable is not ‘greeting’:

$ export INFO="Run make help to show

These sorts of small inconsistencies really break the flow of the entire piece.

Yankee Maharjan • Dec 27 '20

Made the tweaks. On regards for the @ symbol, that's the standard syntax for Makefile and just how it works to disable printing the recipe to your stdout. As the note section suggests, you can try to build a rule where there is no @ symbol in the recipe to see what it means.

If you find the @ symbol weird in front of the bash commands. you can run the commands with -s flag

$ make -s <target-name>

to disable printing the recipe to your stdout.

Hope it helps. Cheers!

Mark Smith • Dec 27 '20

It’s not clear to me what this means:

printing the recipe to your stdout

Are you saying that adding the @ causes make to suppress command output to stdout? Or is it that make doesn’t print the step name to stdout? Or something else?

It might be more obvious with some example make output.