Ease your development hurdles with these [M]ake recipes

Table of contents

1. Introduction
2. Some Recipes/Tasks for your backend projects
   1. Build the project
   2. Clean up the workspace
   3. Run the project
   4. Prepare the local environment for development
   5. Reset the database
   6. Freeze and unfreeze the state of the database
   7. Update non-local environments
   8. File generation scripts
   9. Run arbitrary command
   10. Migrate database
   11. Check code
   12. Bump version
   13. Run unit/acceptance tests
   14. Open the profiling tool
3. Conclusions

Introduction

Make and other similar expert/automation systems like PyInvoke (I highly love this one over other build systems, even over Makefile), Rake, Cake, etc... avoid the necessity of having a bunch of scripts (bash, python...) to perform certain repetitive tasks. Even then, you often chain them in succession to achieve a certain goal (e.g. you know you have to run script_1.sh before script_2.sh, but that dependency is probably not made explicit in the project file structure). These dependencies are difficult to remember as they become more and more complex.

Or you might install a whole bunch of other tools for your project: poetry, gulp or docker just to name a few. Eventually recalling the arguments required to perform a certain action becomes difficult. But it's never difficult to recall the action you need to perform.

Dumb metaphor: It's easier to remember that you need an spanish omelette than remember that you need eggs, potatoes, onions and salt to build such delicious dish. Wouldn't it be easier if you had the power to imagine an omelette and PUFF!, it materializes in front of you, than building the omelette by yourself each and every time? That's where make and other derivatives can help you.

Here I present you with several suggested recipes that I use on a daily basis on the backend projects I have been working. The example are expressed in Makefile syntax, but the very idea can be applied to every build system.

Some Recipes/Tasks for your backend projects

Build the project (make build)

Pretty self explanatory. Every project of a reasonable size and maturity needs to be built, either by compilation, building the docker image, you name it. Therefore, the responsibility of this recipe should be to build/update the project and leave it in a runnable state. Optionally it could run migrations as well to keep the database schema up to date.

Make recipe:

build:
        # Put here the commands to compile, install dependencies, build docker images...

Invocation:

make build

Clean up the workspace (make clean)

This one is the inverse of build. Its responsibility is to clean up the docker images, volumes, generated files... created by the build command.

Make recipe:

clean:
        # Clean docker images, dist files, etc...

Invocation:

make clean

Run the project (make run)

This one runs the project. Prior to spinning up the application (running docker-compose up, executing the main script, etc.), it could prepare the environment to ensure that the project starts up successfully: invalidating caches, creating an AWS session, running database migrations, etc...

This is the command we tell the frontend team to run after building the application with make init to spin up the web server for theirs to run their HTTP requests.

Make recipe:

run:
        # Run your project here

stop:
        # In the same sense that there is a run, it might be interesting for you to have a stop command that kills the process/es that starts up the `run` command.

Invocation:

make run
make stop

Prepare the local environment for development (make setup-dev)

This one is intended for those that are going to do development over the project. It installs dependencies locally, set up pre-commit hooks, etc... basically setup and install anything required for local development.

Make recipe:

setup-dev:
        # Run pre-commit scripts, install dependencies locally...

Invocation:

make setup-dev

Reset the database (make reset)

Eventually and through your testing or the testing of your colleague developers, the database is going to be heavily polluted, and this command should reset the database to factory settings (destroying and creating the local database, re-running the initial data migrations, etc.)

Make recipe:

reset: down
        # Run db scripts to purge the DB/s, remove docker volumes...

Invocation:

make reset

Freeze and unfreeze the state of the database (make freeze and make unfreeze)

I came up with these after some people of my team suggested that it would be cool, from a testing perspective, if we could save a snapshot of the database and then restore it. It proved very useful.

Make recipe:

# Example with postgres, leveraging docker

freeze: down
        docker run -v my_postgres_vol:/volume -v /tmp:/backup --rm loomchild/volume-backup backup postgres

unfreeze: down
    docker run -v my_postgres_vol:/volume -v /tmp:/backup --rm loomchild/volume-backup restore postgres

Invocation:

make freeze
make unfreeze

Update non-local environments like DEV/TEST/UAT/PROD (make update-remote)

Often non-local environments have different characteristics than your machine (they might not run via docker, for example). Having a recipe to handle migrations and updates to those environments is very useful to avoid forgetting to run something.

Make recipe:

update-remote:
        # call here the commands that are intended to be called when you deploy to your remote environment: Database migrations, cache invalidations, scripts to fix something...

Invocation:

make update-remote

File generation scripts (make generate-XXXX)

Sometimes you might need to generate files in your code (documentation, protobuf code, etc...). These kind of commands are intended for such tasks.

Make recipe:

generate-docs:
        # call your doc generation tool (doxygen, sphynx...)

Invocation:

make generate-docs

Run arbitrary command (make run-command)

I work with Django (but this probably applies to other frameworks as well). And wrapping these commands into a recipe is often a good idea to keep the execution centralized.

Make recipe:

run-command:
        # e.g. python src/manage.py ($cmd)

Invocation:

make run-command cmd="XXXXX"

Migrate database (make migrate)

This recipe usually is a dependency of another of the previously mentioned recipes like init and up. As the name implies, it applies the migrations to the database.

Make recipe:

migrate:
        # Run here the command to execute the migrations

Invocation:

make check

Check code (make check)

Any serious project run some kind of code formatting tool to comply with your company coding guidelines and best practices. This command performs these checks on the code.

Make recipe:

check:
    # Example leveraging pre-commit
    pre-commit run --all-files

Invocation:

make check

Bump version (make bump-version LEVEL=patch|minor|major)

This recipe updates the version of your project. Each time I commit code the pre-commit hook calls this command with the patch argument, and each time I deploy to production I manually call it with the minor|major argument depending on the type of code change I'm conducting.

Make recipe:

bump-patch:
    # Example with bump2version
    poetry run bump2version $(LEVEL)
    git add _version.py
    git add .bumpversion.cfg

Invocation:

make bump-patch LEVEL=patch

Run unit/acceptance tests (make run-tests and make run-acceptancetests)

This recipe runs the tests of my application with some default parameters I like to establish. Sometimes I create other similar recipes to run these very tests but with some quirks, like generating coverage and result reports when running.

Make recipe:

run-tests:
        # Call here the command to execute your tests: py.test, NUnit... just make sure that the environment variables needed by your tests are loaded. This is why I like to run my tests with docker compose!

Invocation:

make run-tests

Open the profiling tool (make open-profiling)

You might have been profiling some code to find bottlenecks in your application and generated a report you'd like to visualize with tools like snakeviz or pyprof2calltree. This command will open the latest report generated, ready for visualization.

Make recipe:

open-profiling:
        # Example of command, adjust to your preferred visual tool
    pyprof2calltree -i profiling/results/$(FILE).prof -k

Invocation:

make open-profiling FILE=results

Conclusions

I exposed several recipes that I found useful when developing. Even though I gave some clues of the tooling you could potentially use, it's impossible to provide a detailed implementation since it wildly depends on the programming language you use, the environment, etc... However, what's really important about each recipe is the main objective it tries to achieve, and that's a fact (building the project, running tests...).

I'd like to hear from other fellow developers what other recipes you have come up with that could make our lives a little bit easier. If you have any to share please, feel free to contribute in the comments section!