DEV Community: Daniel Bradley

Better Builds with Make: A Beginner's Guide

Daniel Bradley — Mon, 14 Aug 2023 11:16:12 +0000

If you've ever found yourself repeatedly typing commands to build and manage your projects, it might be time to explore a build automation tool like Make. In this article, we'll cover the basics of Make, which can significantly streamline your development process.

Introduction

The evolution of a project's build process often follows a familiar pattern:

Initial Setup: Dump all your code into a repository.
Basic Instructions: Add a README with rough build instructions.
Script Overload: Include scripts for common tasks.
Build Bottlenecks: Encounter slow builds due to unnecessary rebuilds.
Enter Build Systems: Switch to a more efficient build system like Make.

Let's dive into Make and how it can make your builds better.

Understanding Make's Concepts

Targets and Rules

In the world of Make, a target represents a desired outcome, like a file you want to create. A rule defines how to fulfill a target. Unlike some other build tools that use the concept of "tasks", Make focuses on generating files from other files.

Dependencies and Execution

Make is clever when deciding whether to rebuild a target or not. It compares the timestamps of files to determine if a target needs an update. It rebuilds a target when either:

The target file doesn't exist.
The target file is older than any of its dependencies.

If a target isn't a real file, it's considered a "phony" target, and Make will always run its associated rule.

Rule Syntax

target: prerequites
    commands

Rules are defined as:

Target: file path, followed by a colon.
Prerequisites: files or targets required before running this rule, followed by a newline.
Commands: Shell commands to be executed, indented by a tab.

Variables

Variables in Make help you avoid repetition. You can use variables to store values, filenames, or even results from commands. Variables can be used in target names, prerequisites and commands.

Variables are assigned using an equals sign (=) and their names are case-sensitive.

Common expressions are:

Simple values.
Listing files using the wildcard function.
Calling a shell script (e.g. cat or find).

project = digital-tortoise
src = $(wildcard src/*)
assets = $(shell cat assets.txt)

Running `make`

Run make [target] to build a specific target.

You can specify multiple targets, separated by spaces.
If you don't specify a target, the first rule in the makefile is the default target.

Example: Compiling and Running a Program

Let's work through a basic example to solidify your understanding. Imagine you have a simple program written in a src folder that you want to compile and run to generate an output file.

# Our default rule runs the "build" and "generate" rules
all: build generate

# Friendly aliases for running specific commands
build: bin/my-program
run: out/result.txt

# Tell make that the "all", "build" and "run" targets are 
# "phony" and aren't really files or folders.
.PHONY: all build run

# 1st target compiles our src into the bin directory
bin/my-program: src/*
    # A pretend compiler which takes "src" and "out" args
    compile --src src/* --out bin/my-program

# 2nd step uses our program to generate a text output
out/result.txt: bin/my-program
    bin/my-program --write-to out/result.txt

Handling Non-File Targets

Sometimes, you need to handle targets that aren't single files. You can create sentinel files to represent these targets and their last update times.

My preference is to organize all these sentinel files into a single .make folder. This directory should be ignored and not committed by Git.

There's a few tricks we'll use here:

Variables are computed before any rules run, so we create a variable we won't actually use to run a script to ensure our .make folder exists before we try to write files to it.
There's a special variable $@ which is the name of the current target being run. We use this in the last command of our rule to update the timestamp of our sentinel file (.make/test).
We prefix our touch command with @ to avoid printing it to the output.

# Ensure the .make folder exists when starting make
_ := $(shell mkdir -p .make)

# Add a phony target for "test" as an alias
.PHONY: test
test: .make/test

.make/test:
    # Run our testing tool
    run-tests --all
    # Mark .make/test as up-to-date
    @touch $@

Real-World Example: Compiling TypeScript

Let's see how you can use Make to manage a TypeScript project:

# Ensure the .make folder exists when starting make
_ := $(shell mkdir -p .make)
# Find all our source files
SRC := $(wildcard src/*.ts)

# Default to only building
default: build
# Alias dist as build
build: dist
# Alias dist sentinel target
dist: .make/dist
# Alias package output file
pack: bin/package.tgz
# Mark aliases as phony (not real files)
.PHONY: default build dist pack

# Install packages if definitions changed
.make/node_modules: package.json yarn.lock tsconfig.json
    yarn install
    # Mark .make/node_modules as up-to-date
    @touch $@

# Build dist directory from source and packages
.make/dist: $(SRC) .make/node_modules
    yarn tsc --outDir dist
    # Mark .make/dist as up-to-date
    @touch $@

# Build package zip from dist directory
bin/package.tgz: .make/dist
    cd dist && tar --gzip -cf bin/package.tgz .

This Makefile installs dependencies, compiles TypeScript to JavaScript, and packages the result.

Tips and Recap

Use Meaningful Targets: Name targets by their output file.
Use Sentinel Files: For non-file targets, create sentinel files.
Leverage Dependencies: Model dependencies using file targets.
Provide Phony Targets: Create helpful phony targets for easier use.

Conclusion

Make is a powerful tool to streamline your build process. By understanding its concepts and syntax, you can significantly improve your development workflow.

For an extended guide to makefiles I'd recommend downloading the free Modern Make Handbook.

For a deeper dive, consult the official Make documentation. Happy building!

Appendix: Common Pitfalls To Avoid

Each command is run in its own shell by default. This means that setting an environment variable won't work unless you chain the commands together with &&.
The wildcard function isn't recursive. Use $(shell find ...) as an alternative.
Beware of version-specific features. There's lots of fancy features in the docs, but MacOS ships with a very old version (3.81 from 2006) so won't work out of the box. Either stick to the basic features or ask MacOS users to install a newer version.
Don't use directories as targets.
Don't use committed files as targets.

Why avoid using directories as targets?

Using a directory as a target is generally fine if you're only creating the directory itself. However, if you're also creating files within that directory, the issue arises. The problem is that the "atomic" nature of the target, which ensures it's either fully updated or not, is compromised when dealing with directories.

Targets are considered "atomic" based on timestamps, which determine if something is up-to-date. To maintain this atomicity, timestamps should only be updated after the entire target is successfully completed. When you create a directory as part of the target and it's successful, the timestamp becomes up-to-date. If subsequent commands fail after the directory creation (such as when writing files), the target remains up-to-date despite the failure, and it won't be rerun.

Assuming the directory already exists and updating the timestamp at the end could lead to problems if the directory is accidentally deleted. Manually recreating the directory would give it a newer timestamp, causing the target not to run. While you can use the force option in "make" to work around this, it's not an ideal user experience compared to using sentinel files.

Why avoid using committed files as targets?

The issue with using committed files as targets stems from the expectation that committed files are manually maintained sources. This is the same problem as the question "Why shouldn't target files be edited?" When committed files are treated as targets, it can create confusion when version control updates the file itself and therefore treats the files as already up-to-date after a checkout.

Replacing Build Servers With Pulumi + AWS

Daniel Bradley — Tue, 11 Feb 2020 09:00:50 +0000

Tired of restarting your Jenkins box because something's broken?
Don't like configuring your builds using a slow web UI?
Ditch your flaky Jenkins box and use AWS CodeBuild configured via Pulumi!

Here's the plan, it's a little inception-y, so hang tight...

Create a GitHub repository
Describe an AWS CodeBuild project using TypeScript that will watch itself
Deploy the the infrastructure using Pulumi
Watch it deploy itself as we push changes!

Don't panic, it should become clearer as we get into the code!

Introducing The Tools

AWS CodeBuild - Build and test code with continuous scaling. Pay only for the build time you use.

CodeBuild is very similar to many other build services available but has the added benefit of being tightly integrated into the AWS ecosystem e.g. billing, permissions, automation.

We'll also be using the AWS CLI, so go install that if you've not already got it.

Pulumi - Modern infrastructure as code using real languages.

Sign up for an account - it's free to use for personal use. The account will manage the state of your project deployments.

I'll be using TypeScript here, but you're also able to use a number of other polular languages to achive the same result with Pulumi.

Pulumi Project Setup

Follow through the Pulumi getting started guide for AWS to install the CLI tools, configure your environment and create a blank aws-typescript project called build-setup.

Note: As an example we'll pretend we're pushing it to GitHub at https://github.com/danielrbradley/build-setup.

You should now have an index.ts file with something that looks like this.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("my-bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

The First CodeBuild Project

Delete the lines that created the S3 bucket and exported the name of the bucket - we don't need them.

Here's what we need to setup a CodeBuild project:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const buildProject = new aws.codebuild.Project('build-setup', {
  serviceRole: 'TODO',
  source: {
    type: 'GITHUB',
    location: 'https://github.com/danielrbradley/build-setup.git',
  },
  environment: {
    type: 'LINUX_CONTAINER',
    computeType: 'BUILD_GENERAL1_SMALL',
    image: 'aws/codebuild/standard:3.0',
  },
  artifacts: { type: 'NO_ARTIFACTS' },
});

Let's break this down line-by-line:

const buildProject = new aws.codebuild.Project('build-setup' This creates us a new Pulumi resource representing a CodeBuild project. Creating this doesn't immediately create the resource in AWS but describes to Pulumi what we will want to deploy in the future.
serviceRole: 'TODO' We'll skip over this right now and fix it below.
source: {...} - where should CodeBuild get the source code to build? We're using GitHub, but you can also use other sources too.
environment: {...} What kind of computer do you need for running your build - Linux or Windows, small & cheap or more powerful, the operating system (a docker image)
artifacts Where should any output files be written? This first build won't have any.

Permissions

Back to that serviceRole property. When the build is run, the role we specify here defines what in our AWS account is made accessed to the build job. Because we're running the build inside AWS, we don't need to use access keys, it inherits all the access of the role.

Create a new role for your build to run as. Add this before your CodeBuild project.

const buildRole = new aws.iam.Role('build-setup-role', {
  assumeRolePolicy: {
    Version: '2012-10-17',
    Statement: [
      {
        Effect: 'Allow',
        Principal: {
          Service: 'codebuild.amazonaws.com',
        },
        Action: 'sts:AssumeRole',
      },
    ],
  },
});

This defines a role to be created and specifies through the "assume role policy" that only the CodeBuild service is allowed to use this role.

Now, we need to specify what CodeBuild will be allowed to do when acting as this role. There's two ways we can do this: define our own "inline policy" listing specific services, actions and resources; or attach an existing policy to the role. Here we'll go for the latter:

new aws.iam.RolePolicyAttachment('build-setup-policy', {
  role: buildRole,
  policyArn: 'arn:aws:iam::aws:policy/AdministratorAccess',
});

Important note: To keep the example simple and concise, we're just going to give the role administrator access. I would exercise caution in using this exact approach as it means that anyone who can pushes code to your GitHub repository can change absolutely anything in your AWS account.

Reading line-by-line:

new aws.iam.RolePolicyAttachment('build-setup-policy' We're creating a resource which 'attaches' a policy to a role and giving that attachment the name build-setup-policy.
role: buildRole, The role you want to attach to - which is the role created in the previous step. This can either be a role object or a string containing a role ARN.
policyArn: ... The ARN string of the policy to attach. AdministratorAccess is an AWS managed, built-in policy giving complete unrestricted access to your AWS account.

Now update your buildProject, serviceRole property to point to your new buildRole's arn:

const buildProject = new aws.codebuild.Project("build-setup", {
  serviceRole: buildRole.arn,
  //---------- SNIP ----------//
});

Authenticating CodeBuild with GitHub

Go to GitHub and create a "personal access token". When creating the token, you'll need to tick the repo and admin:repo_hook scopes.

Pulumi has built-in configuration and even supports encrypting individual variables within the project. Copy the created access token and, in your command line, run the command:

pulumi config set --secret github-token YOUR_SECRET_PERSONAL_ACCESS_TOKEN

Next, add a credentials resource in your index.ts:

const config = new pulumi.Config();

new aws.codebuild.SourceCredential('github-token', {
  authType: 'PERSONAL_ACCESS_TOKEN',
  serverType: 'GITHUB',
  token: config.requireSecret('github-token'),
});

This will create a new source credential resource with the name 'github-token' containing your GitHub Personal Access Token. The pulumi.Config() class lets us read the config we just saved using your command line, and decrypt the secret's value.

Triggering Builds

If you deployed this now you'd get a build that you could manually start and would build whatever's in your repository. However, it would be more useful if it automatically started building as soon as you pushed new code to GitHub!

To listen for changes from GitHub we need a "webhook". Add the following resource to build on each new commit pushed to master...

new aws.codebuild.Webhook('build-setup-webhook', {
  projectName: buildProject.name,
  filterGroups: [
    {
      filters: [
        {
          type: 'EVENT',
          pattern: 'PUSH',
        },
        {
          type: 'HEAD_REF',
          pattern: 'refs/heads/master',
        },
      ],
    },
  ],
});

Line-by-line again...

Define the resource type to create and give it a name
Pass the name of the CodeBuild project to trigger
Filter to only the events you're interested in: when a commit is pushed to the master branch

Authenticating CodeBuild with Pulumi

For Pulumi to work in an automated environment you need to create a new Pulumi "Access Token". Copy the token and let's use Pulumi's encrypted config again to store it:

pulumi config set --secret pulumi-access-token YOUR_PULUMI_ACCESS_TOKEN

AWS SSM Parameter Store is a great way to store sensitive values like this within your infrastructure. Let's create a resource to hold the secret value:

const pulumiAccessToken = new aws.ssm.Parameter('pulumi-access-token', {
  type: 'String',
  value: config.requireSecret('pulumi-access-token'),
});

We need the access token in the build environment. Let's change the buildProject resource to load the token from the SSM parameter:

const buildProject = new aws.codebuild.Project('build-setup', {
  //---------- SNIP ----------//
  environment: {
    //---------- SNIP ----------//
    environmentVariables: [
      {
        type: 'PARAMETER_STORE',
        name: 'PULUMI_ACCESS_TOKEN',
        value: pulumiAccessToken.name,
      },
    ],
  },
  //---------- SNIP ----------//
});

CodeBuild Build Specification

The final step of configuration is to tell CodeBuild how to build our project.

CodeBuild will automatically look for a file called buildspec.yml at the root of your repository - let's create that now.

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 12
    commands:
      - curl -fsSL https://get.pulumi.com | sh
      - PATH=$PATH:/root/.pulumi/bin
  pre_build:
    commands:
      - npm ci
      - pulumi login --non-interactive
  build:
    commands:
      - pulumi up --non-interactive

Running through the sections line-by-line:

Install the Node.js v12.x runtime
Download and install Pulumi
Make pulumi available on the $PATH
Restore packages using NPM
Log in to Pulumi (uses the PULUMI_ACCESS_TOKEN environment variable)
Run Pulumi deploy

The --non-interactive option is available on all Pulumi CLI commands to ensure that it doesn't prompt for input at any stage which would cause the build to hang and timeout.

Our first deployment

Right, that's all the coding done! Now to do our first deployment.

Open your command line
Run pulumi up
You'll get a preview of what it's about to do, then select "Yes" to continue

That's it!

The deployment should only take a couple of minutes.

Summary

You created a GitHub project containing a TypeScript file which contains the definitions of a CodeBuild project to create.
This CodeBuild project is configured to watch for changes to the GitHub project and re-deploy itself on each change.
Deployed the first version from your local machine.
Now you can add a few lines of code and push it to GitHub to setup whole new build pipelines!

Getting to the point of the first deploy takes some work, but once you're up and running this is a very efficient and elegant process for managing build projects. At work we've been testing this setup for around a year and have 28 projects configured using this method. The feedback from every developer has been overwhelmingly positive compared to our old Jenkins setup.

From here, there's many interesting avenues to explore:

Adding more repositories to build
Testing changes in pull requests
Using CloudWatch and Lambda to monitor builds and alert you to failures
Use CloudWatch scheduled triggers for nightly build tasks
Abstracting the code to reduce the amount of code you have to write for each new GitHub repository you want to build

Would love to hear about where you take this!