DEV Community: Pascal Landau

A primer on GCP Compute Instance VMs for dockerized Apps [Tutorial Part 8]

Pascal Landau — Mon, 24 Apr 2023 12:14:13 +0000

Getting started with the Google Cloud Platform (GCP) to run Virtual Machines (VMs) and prepare them to run dockerized applications.

This article appeared first on https://www.pascallandau.com/ at A primer on GCP Compute Instance VMs for dockerized Apps [Tutorial Part 8]

In the eighth part of this tutorial series on developing PHP on Docker we will take a look on the Google Cloud Platform (GCP) and create a Compute Instance VM to run dockerized applications. This includes:

creating VMs
using a container registry
using a secret manager

All code samples are publicly available in my Docker PHP Tutorial repository on Github. You find the branch with the final result of this tutorial at part-8-gcp-compute-instance-vm-docker.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Create a CI pipeline for dockerized PHP Apps. and the following one is Deploy dockerized PHP Apps to production on GCP via docker compose.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
Set up a GCP project
Create a service account
- Create service account key file
- Configure IAM permissions
Set up the gcloud CLI tool
Set up the Container Registry
- Authenticate docker
- Pushing images to the registry
- Images are stored in Google Cloud Storage buckets
- Pulling images from the registry
Set up the Secret Manager
- Create a secret via the UI
- View a secret via the UI
- Retrieve a secret via the gcloud cli
- Add the secret gpg key and password
Compute Instances: The GCP VMs
- Create a VM
  - General VM settings
  - Firewall and networks tags
  - The role of the service account
  - Adding a public SSH key
  - Define Availability Policies
  - The actual VM creation
- Log into a VM
  - Login via SSH from the GCP UI
  - Login via SSH with your own key from your host machine
  - Login using the Identity-Aware Proxy (IAP) concept
    - Additional notes on IAP
  - Get root permissions
- ssh and scp commands
  - gcloud ssh --command=""
  - gcloud scp
Provision the VM
- Get the secret gpg key and password from the Secret Manager
- Installing docker and docker compose
- Authenticate docker via gcloud
- Pulling the nginx image
- Start the nginx container
Automate via gcloud commands
- Preconditions: Project and Owner service account
- Configure gcloud to use the master service account
- Enable APIs
- Create and configure a "deployment" service account
- Create secrets
- Create firewall rule for HTTP traffic
- Create a Compute Instance VM
- Provisioning
- Deployment
Putting it all together
Cleanup
Wrapping up

Introduction

In the next tutorial we will deploy our dockerized PHP Apps to "production" via docker compose and will create this "production" environment on GCP (Google Cloud Platform). This tutorial serves as a primer on GCP to build up some fundamental knowledge, because we will use the platform to provide all the infrastructure required to run our dockerized PHP application.

In the process, we'll learn about GCP projects as our own "space" in GCP and service accounts as a way to communicate programmatically. We'll start by doing everything manually via the UI, but will also explain how to do it programmatically via the gcloud cli and end with a fully automated script.

The following video shows the overall flow

Your browser does not support the video tag.

The API keys (see service account key files) that I use are
not in the repository, because I would be billed for any usage. I.e. you must create you own project and keys to follow along.

Caution Following the steps outlined in this tutorial will incur costs, because we will create "real" infrastructure. It won't be much (couple of cents), and it will very likely be covered by the free 300$ grant that you get when trying out GCP (or the general unlimited GCP Free Tier ).

But you should still know about that upfront and make sure to shut everything down / delete everything in case you're trying it out yourself. The "safest" way to do so is Shutting down (deleting) the whole project

Set up a GCP project

On GCP, resources are organized under so-called projects. We can create a project via the Create Project UI:

The project ID must be a globally unique string, and I have chosen pl-dofroscra-p for this tutorial (pl => Pascal Landau; dofroscra => Docker From Scratch; p => production).

Create a service account

As a next step, we need a service account that we can use to make API requests, because we don't want to use our "personal GCP account". Service accounts are created via the IAM & Admin > Service Accounts UI:

Create service account key file

In order to use the account programmatically, we also need to create a key file by choosing the "Manage Keys" option of the corresponding service account.

This will open up a UI at

https://console.cloud.google.com/iam-admin/serviceaccounts/details/$serviceAccountId/keys

where $serviceAccountId is the numeric id of the service account, e.g. 109548647107864470967. To create a key:

click "ADD KEY" and select "Create new key" from the drop down menu
- This will bring up a modal window to choose the key type.
select the recommended JSON type and click "Create".
- GCP will then generate a new key pair, store the public key and offer the private key file as download.
download the file and make sure to treat it like any other private key (ssh, gpg, ...) i.e. never share it publicly!

We will store this file in the root of the codebase at gcp-service-account-key.json and add it to the .gitignore file.

Each service account has also a unique email address that consists of its (non-numeric) id and the project id. You can also find it directly in the key file:

$ grep "email" ./gcp-service-account-key.json
  "client_email": "docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com",

This email address is usually used to reference the service account, e.g. when assigning IAM permissions.

Configure IAM permissions

IAM stands for Identity and Access Management (IAM) and is used for managing permissions on GCP. The two core concepts are "permissions" and "roles":

permissions are fine-grained for particular actions, e.g. storage.buckets.create to "Create Cloud Storage buckets"
roles combine a selection of permissions, e.g. the Cloud Storage Admin role has permissions like
- storage.buckets.create
- storage.buckets.get
- etc.
roles are assigned to users (or service accounts)

You can find a full overview of all permissions in the Permissions Reference and all roles under Understanding roles > Predefined roles.

For this tutorial, we'll assign the following roles to the service account "user" docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com:

Storage Admin
- required to create the GCP bucket for the registry and to pull the images on the VM
Secret Manager Admin
- required to retrieve secrets from the Secret Manager
Compute Admin, Service Account User and IAP-secured Tunnel User
- are necessary for logging into a VM via IAP.

Roles can be assigned through the Cloud Console IAM UI by editing the corresponding user.

Caution: It might take some time (usually a couple of seconds) until changes in IAM permissions take effect.

Set up the `gcloud` CLI tool

The CLI tool for GCP is called gcloud and is available for all operating systems.

In this tutorial we are using version 380.0.0 installed natively on Windows via the GoogleCloudSDKInstaller.exe using the "Bundled Python" option.

FYI: As described under Uninstalling the Google Cloud CLI you can find the installation and config directories via

# installation directory
$ gcloud info --format='value(installation.sdk_root)'
C:\Users\Pascal\AppData\Local\Google\Cloud SDK\google-cloud-sdk

# config directory
$ gcloud info --format='value(config.paths.global_config_dir)'
C:\Users\Pascal\AppData\Roaming\gcloud

I will not use my personal Google account to run gcloud commands, thus I'm not using the "usual" initialization process by running gcloud init. Instead, I will use the service account that we created previously and activate it as described under gcloud auth activate-service-account via

gcloud auth activate-service-account docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com --key-file=./gcp-service-account-key.json --project=pl-dofroscra-p

Output

$ gcloud auth activate-service-account docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com --key-file=./gcp-service-account-key.json --project=pl-dofroscra-p
Activated service account credentials for: [docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com]

FYI: Because we are using a json key file that includes the service account ID, we can also omit the id in the command, i.e.

$ gcloud auth activate-service-account --key-file=./gcp-service-account-key.json --project=pl-dofroscra-p
Activated service account credentials for: [docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com]

Set up the Container Registry

We will use docker compose to run our PHP application in the next tutorial part and need to make our docker images available in a container registry. Luckily, GCP offers a Container Registry product that gives us a ready-to-use private registry as part of a GCP project. Before we can use it, the corresponding Google Container Registry API must be enabled:

You find the Container Registry in the Cloud Console UI under Container Registry.

Authenticate docker

Since the Container Registry is private, we need to authenticate before we can push our docker images. The available authentication methods are described in the GCP docu "Container Registry Authentication methods". For pushing images from our local host system, we will use the service account key file that we created previously and run the command shown in the "JSON key file" section of the the docu.

key=./gcp-service-account-key.json
cat "$key" | docker login -u _json_key --password-stdin https://gcr.io

A successful authentication looks as follows:

$ cat "$key" | docker login -u _json_key --password-stdin https://gcr.io
Login Succeeded

Logging in with your password grants your terminal complete access to your account.
For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/

So what exactly "happens" when we run this command? According to the docker logindocumentation

When you log in, the command stores credentials in $HOME/.docker/config.json
on Linux or %USERPROFILE%/.docker/config.json on Windows
[...]

The Docker Engine can keep user credentials in an external credentials store,
such as the native keychain of the operating system.
[...]

You need to specify the credentials store in
$HOME/.docker/config.json to tell the docker engine to use it.
[...]

By default, Docker looks for the native binary on each of the platforms,
i.e. “osxkeychain” on macOS, “wincred” on windows, and “pass” on Linux.

In other words: I won't be able to see the content of the service account key file in "plain text" anywhere but docker will utilize the OS specific tools to store them securely. After I ran the docker login command on Windows, I found the following content in ~/.docker/config.json:

$ cat ~/.docker/config.json
{
        "auths": {
                "gcr.io": {}
        },
        "credsStore": "desktop"
}

FYI: "desktop" seems to be a wrapper for the Wincred executable.

Pushing images to the registry

For this tutorial, we will create a super simple nginx alpine image that provides a "Hello world" hello.html file via

docker build -t my-nginx -f - . <<EOF
FROM nginx:1.21.5-alpine

RUN echo "Hello world" >> /usr/share/nginx/html/hello.html

EOF

The name of the image is my-nginx

$ docker image ls | grep my-nginx
my-nginx         latest             42dd1608d126   50 seconds ago    23.5MB

In order to push an image to a registry,
the image name must be prefixed with the corresponding registry. This was quite confusing to me, because I would have expected to be able to run something like this:

$ docker push my-nginx --registry=gcr.io

unknown flag: --registry
See 'docker push --help'.

But nope, there is no such --registry option. Even worse: Omitting it would cause a push to docker.io, the "default" registry:

$ docker push my-nginx
Using default tag: latest
The push refers to repository [docker.io/my-nginx]

According to the GCP docs on Pushing and pulling images, the following steps are necessary to push an image to a GCP registry:

Tag the image with its target path in Container Registry, including the gcr.io registry host and the project ID my-project

Push the image to the registry

In our case the target path to our Container Registry is

gcr.io/pl-dofroscra-p

because pl-dofroscra-p is the id of the GCP project we created previously.

The full image name becomes

gcr.io/pl-dofroscra-p/my-nginx

To push the my-nginx image, we must first "add another name" to it via docker tag

$ docker tag my-nginx gcr.io/pl-dofroscra-p/my-nginx

$ docker image ls
REPOSITORY                       TAG                IMAGE ID       CREATED          SIZE
my-nginx                         latest             ba7a2c5faf0d   15 minutes ago   23.5MB
gcr.io/pl-dofroscra-p/my-nginx   latest             ba7a2c5faf0d   15 minutes ago   23.5MB

and push that name afterwards

$ docker push gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
The push refers to repository [gcr.io/pl-dofroscra-p/my-nginx]
134174afa9ad: Preparing
cb7b4430c52d: Preparing
419df8b60032: Preparing
0e835d02c1b5: Preparing
5ee3266a70bd: Preparing
3f87f0a06073: Preparing
1c9c1e42aafa: Preparing
8d3ac3489996: Preparing
8d3ac3489996: Waiting
3f87f0a06073: Waiting
1c9c1e42aafa: Waiting
cb7b4430c52d: Pushed
134174afa9ad: Pushed
419df8b60032: Pushed
5ee3266a70bd: Pushed
0e835d02c1b5: Pushed
8d3ac3489996: Layer already exists
3f87f0a06073: Pushed
1c9c1e42aafa: Pushed
latest: digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f size: 1982

You can then find the image in the UI of the Container Registry:

Don't worry: We won't have to do the tagging every time before a push, because we will set up make to use the correct name automatically when building the images in the next part.

Images are stored in Google Cloud Storage buckets

We assigned the Storage Admin role to the service account previously that contains the storage.buckets.create permission. If we wouldn't have done that, the following error would have occurred:

denied: Token exchange failed for project 'pl-dofroscra-p'. Caller does not have permission 'storage.buckets.create'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

The Container Registry tries to store the docker images in a Google Cloud Storage bucket that is created on the fly when the very first image is pushed, see the GCP docs on "Adding a registry":

The first image push to a hostname triggers creation of the registry in a project
and the corresponding Cloud Storage storage bucket.
This initial push requires project-wide permissions to create storage buckets.

You can find the bucket, that in my case is named artifacts.pl-dofroscra-p.appspot.com in the Cloud Storage UI:

CAUTION: Make sure to delete this bucket once you are done with the tutorial - otherwise storage costs will incur.

Pulling images from the registry

According to the GCP docs to pull an image from the Container Registry we need to be authenticated with a user that has the permissions of the Storage Object Viewer role to access the "raw" images. FYI: The Storage Admin role that we assigned previously has all the permissions of the Storage Object Viewer role.

Then use the fully qualified image name as before:

docker pull gcr.io/pl-dofroscra-p/my-nginx

Output if the image is cached

$ docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
Digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f
Status: Image is up to date for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

or if doesn't exist

docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
59bf1c3509f3: Pull complete 
f3322597df46: Pull complete 
d09cf91cabdc: Pull complete 
3a97535ac2ef: Pull complete 
919ade35f869: Pull complete 
40e5d2fe5bcd: Pull complete 
c72acb0c83a5: Pull complete 
d6baa2bee4a5: Pull complete 
Digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f
Status: Downloaded newer image for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

Set up the Secret Manager

Even though we use git secret to manage our secrets, we still need the gpg secret key for decryption. This key is "a secret in itself" and we will use the GCP Secret Manager to store it and retrieve it later from a VM.

It can be managed from the Security > Secret Manager UI once we have enabled the Secret Manager API

Create a secret via the UI

To create a secret:

navigate to the Secret Manager UI and click the "+ CREATE SECRET" button
enter the secret name and secret value in the form (we can ignore the other advanced settings like "Replication policy" and "Encryption" for now)
- FYI: the secret name can only contain English letters (A-Z), numbers (0-9), dashes (-), and underscores (_)
click the "CREATE SECRET" button

The following gif shows the creation of a secret named my_secret_key with the value my_secret_value.

View a secret via the UI

The Security > Secret Manager UI lists all existing secrets. Clicking on a secret will lead you to the Secret Detail UI at the URL

https://console.cloud.google.com/security/secret-manager/secret/$secretName/versions

e.g. for secret name my_secret_key:

https://console.cloud.google.com/security/secret-manager/secret/my_secret_key/versions

The UI shows all versions of the secret, though we currently only have one. To view the actual secret value, click on the three dots in the "Actions" column and select "View secret value" (see gif in the previous section).

Retrieve a secret via the `gcloud` cli

To retrieve a secret with a service account via the gcloud cli, it needs the permission secretmanager.versions.access, that is part of the Secret Manager Secret Accessor role (as well as the Secret Manager Admin role). Let's first show all available secrets via

gcloud secrets list

$ gcloud secrets list
NAME           CREATED              REPLICATION_POLICY  LOCATIONS
my_secret_key  2022-05-15T05:38:11  automatic           -

To "see" the value of my_secret_key, we must also define the corresponding version. All versions can be listed via

gcloud secrets versions list my_secret_key

$ gcloud secrets versions list my_secret_key
NAME  STATE    CREATED              DESTROYED
1     enabled  2022-05-15T05:38:13  -

The actual secret value for version 1 of my_secret_key is accessed via

gcloud secrets versions access 1 --secret=my_secret_key

$ gcloud secrets versions access 1 --secret=my_secret_key
my_secret_value

You can also use the string latest as version to retrieve the latest enabled version

Version resource - Numeric secret version to access or a configured alias
(including 'latest' to use the latest version).

gcloud secrets versions access latest --secret=my_secret_key

$ gcloud secrets versions access latest --secret=my_secret_key
my_secret_value

Add the secret `gpg` key and password

We already know how to create another gpg key pair and add the corresponding email to git secret from when we have set up the CI pipelines (see section Add a password-protected secret gpg key) . We will do the same once more for the production environment via

name="Production Deployment"
email="production@example.com"
passphrase=87654321
secret=secret-production-protected.gpg.example
public=.dev/gpg-keys/production-public.gpg
# export key pair
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
Passphrase: $passphrase
EOF

# export the private key
gpg --output $secret --pinentry-mode=loopback --passphrase  "$passphrase" --armor --export-secret-key $email

# export the public key
gpg --armor --export $email > $public

You can find the secret key in the codebase at secret-production-protected.gpg.example and the public key at .dev/gpg-keys/production-public.gpg.

I have also added the secret gpg key and password as secrets to the secret manager

Compute Instances: The GCP VMs

Compute Instances are the equivalent of AWS EC2 instances.

To run our application, we need a VM with a public IP address so that it can be reached from the internet. VMs on GCP are called Compute Instances and can be managed from the Compute Instance UI - though we must first activate the Compute Instance API.

Create a VM

We can simply create a new instance from the Create an instance UI.

General VM settings

We'll use the following settings:

Name: dofroscra-test
Region us-central1 (Iowa) and Zone us-central1-a
Machine family: General Purpose > E2 > e2-micro (2 vCPU, 1 GB memory)
Boot Disk: Debian GNU/Linux 11 (bullseye); 10 GB
Identity and API access: Choose the "Docker PHP Tutorial deployment account" service account that we created previously
Firewall: Allow HTTP traffic

Firewall and networks tags

Ticking the Allow HTTP traffic checkbox will cause two things when the instance is created:

a new firewall rule named default-allow-http is created that allows incoming traffic from port 80 and is only applied to instances with the network tag http-server. You can see the new rule in the Firewall UI
the network tag http-server is added to the instance, effectively enabling the aforementioned firewall rule for the instance

The role of the service account

The service account that we have chosen in the previous step will be attached to the Compute Instance. I.e. it will be available on the instance and we will have access to the account when we log into the instance. Conveniently, thegcloud cli is pre-installed on every Compute Instance:

If you're using an instance on Compute Engine, the gcloud CLI is installed by default.

The exact rules for what the service account can do are described in the Compute Engine docs for "Service accounts". In short: The possible actions will be constrained by the IAM permissions of the service account and the Access scopes of the Compute Instance which are set as outlined in the "Default scopes" section. Just take this as an aside, we won't have to modify anything here.

We will use the service account later, when we log into the Compute Instance and run docker pull from there to pull images from the registry.

Adding a public SSH key

In addition, I added my own public ssh key

ssh-rsa AAAAB3NzaC1yc2....6row== pascal.landau@MY_LAPTOP

CAUTION: The username for this key will be defined at the end of the key! E.g. in the example above, the username would be pascal.landau. This is important for logging in later via SSH from your local machine.

Define Availability Policies

We will make the instance preemptible, by choosing the VM provisioning model Spot. This makes it much cheaper but GCP "might" terminate the instance randomly if the capacity is needed somewhere else (and definitely after 24 hours). This is completely fine for our test use case. The final costs are ~2$ per month for this instance.

The actual VM creation

Finally, click the "Create" button at the bottom of the page

Once the instance is created, you can see it in the Compute Instances > VM instances UI

Next to the instance name dofroscra-test we can see the external IP address 35.192.212.130 Opening it in a browser via http://35.192.212.130/ won't show anything though, because we didn't deploy the application yet.

CAUTION: Make sure to shutdown and remove this instance by the end of the tutorial - otherwise compute costs will incur.

Log into a VM

There are multiple ways to log into the VM / Compute Instance outlined in Connecting to Linux VMs using advanced methods.

I'm going to describe three of them:

PS: It's worth keeping the Troubleshooting SSH guide as a bookmark.

Login via SSH from the GCP UI

Probably the easiest way to log in: Simply click the "SSH" button in the Compute Instances > VM instances UI next to the instance you want to log in. This will create a web shell that uses an ephemeral SSH key according to the GCP documentation: Connect to Linux VMs > Connect to VMs

When you connect to VMs using the Cloud Console, Compute Engine creates an ephemeral SSH key for you.

Login via SSH with your own key from your host machine

This method is probably closest to what you are used to from working with "other" VMs. In this case, the instance has to be publicly available (i.e. reachable "from the internet") and
expose a port for SSH connections (usually 22). In addition, your public SSH key needs to be deployed.

All of those requirements are true for the Compute Instance that we just created:

the public ip address is 35.192.212.130
port 22 is open by default via the default-allow-ssh firewall rule, see How to Configure Firewall Rules in Google Cloud Platform and the official documentation on Troubleshooting SSH > By default, Compute Engine VMs allow SSH access on port 22.
we added our public SSH key using pascal.landau as the username

So we can now simply login via

ssh pascal.landau@35.192.212.130

or if you need to specify the location of the private key via the -i option

ssh -i ~/.ssh/id_rsa pascal.landau@35.192.212.130

$ ssh pascal.landau@35.192.212.130
Linux dofroscra-test 4.19.0-20-cloud-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Apr 10 16:21:00 2022 from 54.74.228.207
pascal.landau@dofroscra-test:~$

Login using the Identity-Aware Proxy (IAP) concept

Note: This is the preferred way of logging into a GCP VM

To login via IAP we need the gcloud CLI that will use API requests (via HTTPS) under the hood to authenticate the Google user (or in our case: the service account) and then proxy the requests to the VM via GCP's Identity-Aware Proxy (IAP) as described in Connecting through Identity-Aware Proxy (IAP) for TCP and in more detail under Using IAP for TCP forwarding > Tunneling SSH connections.

The corresponding command is gcloud compute ssh, e.g.:

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p

Note the --tunnel-through-iap flag: Without it, gcloud would instead attempt to use a "normal" ssh connection if the VM is publicly reachable as per docu:

If the instance doesn't have an external IP address, the connection automatically uses
IAP TCP tunneling. If the instance does have an external IP address, the connection uses the
external IP address instead of IAP TCP tunneling.

Output

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p
WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The PuTTY PPK SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Updating project ssh metadata...
..............................................Updated [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p].
.done.
Waiting for SSH key to propagate.

Under the hood, this command will automatically create an SSH key pair on your local machine under ~/.ssh/ named google_compute_engine (unless they already exist) and upload the public key to the instance.

$ ls -l ~/.ssh/ | grep google_compute_engine
-rw-r--r-- 1 Pascal 197121 1675 Apr 11 09:25 google_compute_engine
-rw-r--r-- 1 Pascal 197121 1456 Apr 11 09:25 google_compute_engine.ppk
-rw-r--r-- 1 Pascal 197121  420 Apr 11 09:25 google_compute_engine.pub

It also opens a Putty session and logs you into in the instance:

Using username "Pascal".
Authenticating with public key "LAPTOP-0DNL2Q02\Pascal@LAPTOP-0DNL2Q02"
Linux dofroscra-test 4.19.0-20-cloud-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Apr 11 07:22:32 2022 from 54.74.228.207
Pascal@dofroscra-test:~$

Caution: The Putty session will be closed automatically when you abort the original gcloud compute ssh command!

Additional notes on IAP

In order to enable SSH connections via IAP, our service account needs the following IAM roles:

roles/compute.instanceAdmin.v1, role: Compute Admin
roles/iam.serviceAccountUser, role: Service Account User
roles/iap.tunnelResourceAccessor, role: IAP-secured Tunnel User Otherwise, you might run into a couple of errors like:

ERROR: (gcloud.compute.ssh) Could not fetch resource:
 - Required 'compute.instances.get' permission for 'projects/pl-dofroscra-p/zones/us-central1-a/instances/dofroscra-test'

(roles/compute.instanceAdmin.v1 missing)

ERROR: (gcloud.compute.ssh) Could not add SSH key to instance metadata:
 - The user does not have access to service account 'docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com'.  User: 'docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com'.  Ask a project owner to grant you the iam.serviceAccountUser role on the service account

(roles/iam.serviceAccountUser missing)

Remote side unexpectedly closed connection

(roles/iap.tunnelResourceAccessor missing)

Note: This error can also occur if you try to use IAP directly after starting a VM. In this case wait a couple of seconds and try again.

The general "flow" looks like this

Using IAP might look overly complicated at first, but it offers a number of benefits:

we can leverage Google's authentication system and the powerful IAM permission management to manage permissions - no more need to deploy custom SSH keys to the VMs
we don't need a public IP address any longer, see Overview of TCP forwarding > How IAP's TCP forwarding works > A special case, establishing an SSH connection using gcloud compute ssh wraps the SSH > connection inside HTTPS and forwards it to the remote instance without the need of a > listening port on local host. > > [...] > > TCP forwarding with IAP doesn't require a public, routable IP address assigned to your > resource. Instead, it uses internal IPs.

This is nice, because in our final PHP application only the nginx container should be
accessible publicly - php-fpm and the php-workers shouldn't. Usually we would use a

so-called Bastion Host or Jump Box to deal with
this problem, but thanks to IAP we don't have to

Additional resources

Get `root` permissions

The group google-sudoers exists on each GCP Compute Instance. It is configured for passwordless sudo via /etc/sudoers.d/google_sudoers

$ sudo cat /etc/sudoers.d/google_sudoers 
%google-sudoers ALL=(ALL:ALL) NOPASSWD:ALL

I.e. each user added to this group can use sudo without password or simply run sudo -i to become root. Your user should be added automatically to that group. This can be verified with the id command (run on the VM)

Pascal@dofroscra-test:~$ id
uid=1002(Pascal) gid=1003(Pascal) groups=1003(Pascal),4(adm),30(dip),44(video),46(plugdev),1000(google-sudoers)

# => 1000(google-sudoers)

`ssh` and `scp` commands

It is quite common to copy files from / to a VM and to run commands on it. This is usually done via ssh and scp. The gcloud cli offers equivalent commands that can also make use of the Identity-Aware Proxy (IAP) concept:

`gcloud ssh --command=""`

Documentation: gcloud ssh --command

Example:

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="whoami"

Output:

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="whoami"
Pascal

`gcloud scp`

Documentation: gcloud scp

Example:

gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt ./local-file2.txt dofroscra-test:tmp/

# ---
echo "1" >> ./local-file1.txt
echo "2" >> ./local-file2.txt
gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="rm -rf ~/test; mkdir -p ~/test"
gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"
gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt ./local-file2.txt dofroscra-test:test/
gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"

Output:

$ echo "1" >> ./local-file1.txt
$ echo "2" >> ./local-file2.txt
$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"
total 0

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt ./local-file2.txt dofroscra-test:test/
local-file1.txt           | 0 kB |   0.0 kB/s | ETA: 00:00:00 | 100%
local-file2.txt           | 0 kB |   0.0 kB/s | ETA: 00:00:00 | 100%

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="ls -l ~/test"
total 8
-rw-r--r-- 1 Pascal Pascal 4 Jun  2 13:06 local-file1.txt
-rw-r--r-- 1 Pascal Pascal 4 Jun  2 13:06 local-file2.txt

Note: According to the examples in the documentation, it should also be possible to use the ~ to define that the destination on the remote VM is relative to the home directory. However, this did not work for me, as I kept getting the error

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt dofroscra-test:~/test
pscp: remote filespec ~/test: not a directory

even though the diretory existed. But removing the ~/ part worked

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./local-file1.txt  dofroscra-test:test/
local-file1.txt           | 0 kB |   0.0 kB/s | ETA: 00:00:00 | 100%

Provision the VM

Get the secret `gpg` key and password from the Secret Manager

Before we shift our attention to docker, let's quickly deal with the secrets. We won't need them for this part of the tutorial. but will need the secret gpg key and its password to decrypt the secrets in the php containers in the next part. To recap: We have added the them previously and can now retrieve them as explained under section Retrieve a secret via the gcloud cli via

gcloud secrets versions access 1 --secret=GPG_KEY > secret.gpg

GPG_PASSWORD=$(gcloud secrets versions access 1 --secret=GPG_PASSWORD)

$ gcloud secrets versions access 1 --secret=GPG_KEY > secret.gpg
$ head secret.gpg
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQPGBGKA1psBCACq5zYDT587CVZEIWXbUplfAGQZOQJALmzErYpTp0jt+rp4vJhR
U5xahy3pqCq81Cnny5YME50ybB3pW/WcHxWLBDo+he8PKeLbp6wFFjJns+3u4opH
9gFMElyHpzTGiDQYfx/CgY2hKz7GSqpjmnOaKxYvGv0EsbZczyHY1WIN/YFzb0tI
tY7J4zTSH05I+aazRdHyn28QcCRcIT9+4q+5Vk8gz8mmgoqVpyeNgQcqJjcd03iP
WUZd1vZCumOvdG5PZNlc/wPFhqLDmYyLmJ7pt5bWIgty9BjYK8Z2NOdUaekqVEJ+
r29HbzwgFLLE2gd52f07h2y2YgMdWdz4FDxVABEBAAH+BwMC9veBYT2oigXxExLl
7fZKVjw02lEr1NpYd5X1ge9WPU/1qumATJWounzciiETpsYGsbPd9zFRJP4E3JZl
sFSh4p0/kXYTuenYD8wgGkeYyN4lm53IHfqSn2z9JMW5Kz9XEODtKJl8fjcn9Zeb

$ GPG_PASSWORD=$(gcloud secrets versions access 1 --secret=GPG_PASSWORD)
$ echo $GPG_PASSWORD
87654321

Installing `docker` and `docker compose`

Since we created the VM with a Debian OS, we'll follow the official Debian installation instructions for Docker Engine and run the following commands while we are logged into the VM:

# install required tools
sudo apt-get update -yq && apt-get install -yq \
     ca-certificates \
     curl \
     gnupg \
     lsb-release

# add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# set up the stable repository
echo \
 "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
 $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# install Docker Engine
sudo apt-get update -yq && sudo apt-get install -yq \
     docker-ce \
     docker-ce-cli \
     containerd.io \
     docker-compose-plugin

I have also added these instructions to the script .infrastructure/scripts/provision.sh.

Afterwards we check via

docker --version
docker compose version

if docker and docker compose are available

$ docker --version
Docker version 20.10.15, build fd82621

$ docker compose version
Docker Compose version v2.5.0

Authenticate docker via `gcloud`

I recommend running the commands as root via

sudo -i

pascal_landau@dofroscra-test:~$ sudo -i
root@dofroscra-test:~#

Otherwise, we might run into docker permission errors like

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/create?fromImage=...": dial unix /var/run/docker.sock: connect: permission denied

FYI: As an alternative we could also add the non-root user to the docker group via

user=$(whoami)
sudo usermod -a -G docker $user

$ user=$(whoami)
$ sudo usermod -a -G docker $user
$ id
uid=1001(pascal_landau) gid=1002(pascal_landau) groups=1002(pascal_landau),4(adm),30(dip),44(video),46(plugdev),997(docker),1000(google-sudoers)

Locally, we could use the JSON key file of the service account for authentication, but we don't have access to that key file on the VM. Instead, we will authenticate docker via the pre-installed gcloud cli and the attached service account as described in the GCP docs for the Container Registry Authentication methods under section "gcloud credential helper" via

gcloud auth configure-docker --quiet

Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.

This creates the file /root/.docker/config.json that we already encountered when we authenticated docker locally to push images. In this case it has the following content

$ cat /root/.docker/config.json
{
  "credHelpers": {
    "gcr.io": "gcloud",
    "us.gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud"
  }
}

The creadHelpers are described in the docker login docs

Credential helpers are similar to the credential store above, but act as the designated
programs to handle credentials for specific registries. The default credential store
(credsStore or the config file itself) will not be used for operations concerning
credentials of the specified registries.

In other words:

we are using gcr.io as registry
this registry is defined to use the credential helper gcloud
i.e. it will use the pre-initialized gcloud cli for authentication

Pulling the `nginx` image

Once we are authenticated, we can simply run docker pull with the full image name to retrieve the nginx image that we pushed previously

docker pull gcr.io/pl-dofroscra-p/my-nginx

$ docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
59bf1c3509f3: Pull complete 
f3322597df46: Pull complete 
d09cf91cabdc: Pull complete 
3a97535ac2ef: Pull complete 
919ade35f869: Pull complete 
40e5d2fe5bcd: Pull complete 
c72acb0c83a5: Pull complete 
d6baa2bee4a5: Pull complete 
Digest: sha256:0740591fb686227d8cdf4e42b784f634cbaf9f5caa6ee478e3bcc24aeef75d7f
Status: Downloaded newer image for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

If we wouldn't be authenticated, we would run into the following error

$ docker pull gcr.io/pl-dofroscra-p/my-nginx
Using default tag: latest
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Start the `nginx` container

For now, we will simply run the nginx container with docker run via

docker run --name nginx -p 80:80 --rm -d gcr.io/pl-dofroscra-p/my-nginx:latest

give it the name nginx so we can easily reference it later via --name nginx
map port 80 from the host to port 80 of the container so that HTTP requests to the VM are handled via the container via -p 80:80, see Docker docs on "Publish or expose port (-p, --expose)"
make it run in the background via -d (--detach) and remove it after shutdown automatically via -rm (--remove)

Output

root@dofroscra-test:~# docker run -p 80:80 -d --name nginx gcr.io/pl-dofroscra-p/my-nginx:latest
dd49bedad97c06f698d06a140c5091c04ad81b2f75632e222927e7f71cf28c18

root@dofroscra-test:~# docker logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/05/08 12:01:43 [notice] 1#1: using the "epoll" event method
2022/05/08 12:01:43 [notice] 1#1: nginx/1.21.5
2022/05/08 12:01:43 [notice] 1#1: built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027) 
2022/05/08 12:01:43 [notice] 1#1: OS: Linux 4.19.0-20-cloud-amd64
2022/05/08 12:01:43 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/05/08 12:01:43 [notice] 1#1: start worker processes
2022/05/08 12:01:43 [notice] 1#1: start worker process 30
2022/05/08 12:01:43 [notice] 1#1: start worker process 31

We're all set to serve HTTP requests: The requests will be passed to the nginx container due to the mapped port 80, i.e. we can simply use the public IP address 35.192.212.130 in a curl request and check the "Hello world" file we've added to the image:

$ curl -s http://35.192.212.130/hello.html
Hello world

Automate via `gcloud` commands

So far we have mostly used the UI to configure everything. This is great for understanding how things work, but it's not so great for maintainability:

UIs change
"Clicking" stuff takes quite some time
we can't automate anything

Fortunately, we can also use the gcloud cli for most of the tasks.

Preconditions: Project and `Owner` service account

I'll assume that

gcloud is installed
a GCP project exists

In addition, I'll manually create a new "master" service account and assign it the role Owner. We'll use that service account in the gcloud cli to enable all the APIs and manage the resources.

FYI: We could also do this with our personal GCP user, but I plan to use terraform in a later tutorial which will require a service account anyway.

We will use docker-php-tutorial-master as service account ID and store the key file of the account at the root of the codebase under gcp-master-service-account-key.json (which is also added to the .gitignore file).

Configure `gcloud` to use the master service account

Run

gcloud auth activate-service-account --key-file=./gcp-master-service-account-key.json --project=pl-dofroscra-p

$ gcloud auth activate-service-account --key-file=./gcp-master-service-account-key.json --project=pl-dofroscra-p
Activated service account credentials for: [docker-php-tutorial-master@pl-dofroscra-p.iam.gserviceaccount.com]

Enable APIs

APIs can be enabled via gcloud services enable $serviceName, (see also the Docu on "Enabling an API in your Google Cloud project").

The $serviceName is shown on the API overview page of a service, see the following example for the Container Registry

We need the APIs for

Container Registry: containerregistry.googleapis.com
Secret Manager: secretmanager.googleapis.com
Compute Engine: compute.googleapis.com
IAM: iam.googleapis.com
Cloud Storage: storage.googleapis.com
Cloud Resource Manager: cloudresourcemanager.googleapis.com

gcloud services enable containerregistry.googleapis.com secretmanager.googleapis.com compute.googleapis.com iam.googleapis.com storage.googleapis.com cloudresourcemanager.googleapis.com

$ gcloud services enable containerregistry.googleapis.com secretmanager.googleapis.com compute.googleapis.com iam.googleapis.com storage.googleapis.com cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-386551299607-87333b29-c7eb-40b8-b951-8c86185bbf49" finished successfully.

Create and configure a "deployment" service account

We won't use the master Owner service account for the deployment but create a custom one with only the necessary permissions.

Service accounts are created via gcloud iam service-accounts create $serviceAccountId, (see also the Docu on "Creating and managing service accounts").

The $serviceAccountId is the id of the service account, e.g. docker-php-tutorial-deployment in our previous example. In addition, we can define a --description and a --display-name.

gcloud iam service-accounts create docker-php-tutorial-deployment \
  --description="Used for the deployment of the Docker PHP Tutorial application" \
  --display-name="Docker PHP Tutorial Deployment Account"

$ gcloud iam service-accounts create docker-php-tutorial-deployment \
>   --description="Used for the deployment of the Docker PHP Tutorial application" \
>   --display-name="Docker PHP Tutorial Deployment Account"
Created service account [docker-php-tutorial-deployment].

Then, we need a key file for authentication. It can be created via gcloud iam service-accounts keys create $localPathToKeyFile --iam-account=$serviceAccountEmail (see also the Docu on "Create and manage service account keys").

The $localPathToKeyFile is used to store the key file locally, e.g. gcp-service-account-key.json in our previous example and $serviceAccountEmail is the email address of the service account. It has the form

$serviceAccountId@$projectId.iam.gserviceaccount.com

e.g.

docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com

Example:

gcloud iam service-accounts keys create ./gcp-service-account-key.json \
  --iam-account=docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com

$ gcloud iam service-accounts keys create ./gcp-service-account-key.json \
>   --iam-account=docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
created key [810df0c6df21de44dc5e431d2b569d74555ba3f9] of type [json] as [./gcp-service-account-key.json] for [docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com]

Finally, we must assign the required IAM roles via gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=$roleId, (see also the Docu on "Manage access to projects, folders, and organizations").

The $projectName is the name of the GCP project, $serviceAccountEmail the email address from before and the $roleId the name of the role as listed under Understanding roles. In our case that's:

Storage Admin: roles/storage.admin
Secret Manager Admin: roles/secretmanager.admin
Compute Admin: roles/compute.admin
Service Account User: roles/iam.serviceAccountUser
IAP-secured Tunnel User: roles/iap.tunnelResourceAccessor

projectName="pl-dofroscra-p"
serviceAccountEmail="docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com"

gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/storage.admin
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/secretmanager.admin
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/compute.admin
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/iam.serviceAccountUser
gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/iap.tunnelResourceAccessor

$ projectName="pl-dofroscra-p"
$ serviceAccountEmail="docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com"
$ gcloud projects add-iam-policy-binding $projectName --member=serviceAccount:$serviceAccountEmail --role=roles/storage.admin
Updated IAM policy for project [pl-dofroscra-p].
bindings:
- members:
  - serviceAccount:service-386551299607@compute-system.iam.gserviceaccount.com
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:service-386551299607@containerregistry.iam.gserviceaccount.com
  role: roles/containerregistry.ServiceAgent
- members:
  - serviceAccount:386551299607-compute@developer.gserviceaccount.com
  - serviceAccount:386551299607@cloudservices.gserviceaccount.com
  role: roles/editor
- members:
  - serviceAccount:docker-php-tutorial-master@pl-dofroscra-p.iam.gserviceaccount.com
  - user:pascal.landau@gmail.com
  role: roles/owner
- members:
  - serviceAccount:service-386551299607@gcp-sa-pubsub.iam.gserviceaccount.com
  role: roles/pubsub.serviceAgent
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/storage.admin
etag: BwXgKxHg7gA=
version: 1

# ...

Updated IAM policy for project [pl-dofroscra-p].
bindings:
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/compute.admin
- members:
  - serviceAccount:service-386551299607@compute-system.iam.gserviceaccount.com
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:service-386551299607@containerregistry.iam.gserviceaccount.com
  role: roles/containerregistry.ServiceAgent
- members:
  - serviceAccount:386551299607-compute@developer.gserviceaccount.com
  - serviceAccount:386551299607@cloudservices.gserviceaccount.com
  role: roles/editor
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/iam.serviceAccountUser
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/iap.tunnelResourceAccessor
- members:
  - serviceAccount:docker-php-tutorial-master@pl-dofroscra-p.iam.gserviceaccount.com
  - user:pascal.landau@gmail.com
  role: roles/owner
- members:
  - serviceAccount:service-386551299607@gcp-sa-pubsub.iam.gserviceaccount.com
  role: roles/pubsub.serviceAgent
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/secretmanager.admin
- members:
  - serviceAccount:docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/storage.admin
etag: BwXgKxm0Vtk=
version: 1

Create secrets

Secrets are created via gcloud secrets create $secretId", (see also the Docu on "Creating and accessing secrets").

The $secretId is the name of the secret, e.g. my_secret_key in our previous example, and we must also add a new version with the value of the secret via gcloud secrets versions add $secretId --data-file="/path/to/file.txt".

gcloud secrets create my_secret_key

echo -n "my_secret_value" | gcloud secrets versions add my_secret_key --data-file=-

$ gcloud secrets create my_secret_key
Created secret [my_secret_key].

$ echo -n "my_secret_value" | gcloud secrets versions add my_secret_key --data-file=-
Created version [1] of the secret [my_secret_key].

We also need to do the same for the gpg secret key and its password:

gcloud secrets create GPG_KEY
echo gcloud secrets versions add GPG_KEY --data-file=secret-production-protected.gpg.example

gcloud secrets create GPG_PASSWORD
echo -n "87654321" | gcloud secrets versions add GPG_PASSWORD --data-file=-

$ gcloud secrets create GPG_KEY
Created secret [GPG_KEY].
$ gcloud secrets versions add GPG_KEY --data-file=secret-production-protected.gpg.example
Created version [1] of the secret [GPG_KEY].

$ gcloud secrets create GPG_PASSWORD
Created secret [GPG_PASSWORD].
$ echo -n "87654321" | gcloud secrets versions add GPG_PASSWORD --data-file=-
Created version [1] of the secret [GPG_PASSWORD].

Create firewall rule for HTTP traffic

Firewall rules can be created via gcloud compute firewall-rules create $ruleName (see also the Docu on "Using firewall rules").

We'll stick to the same conventions that have been used when creating the VM via the UI by using default-allow-http as the $ruleName and http-server as the network tag (via the --target-tags option)

 gcloud compute firewall-rules create default-allow-http --allow tcp:80 --target-tags=http-server

$ gcloud compute firewall-rules create default-allow-http --allow tcp:80 --target-tags=http-server
Creating firewall...
..Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/global/firewalls/default-allow-http].
done.
NAME                NETWORK  DIRECTION  PRIORITY  ALLOW   DENY  DISABLED
default-allow-http  default  INGRESS    1000      tcp:80        False

Create a Compute Instance VM

Compute Instances can be created via gcloud compute instances create $vmName (see also the Docu on "Creating and starting a VM instance").

The $vmName defines the name of the VM, e.g. dofroscra-test in the previous example. In addition, there are a lot of options to customize the VM, e.g.

--image-family and --image-project
- define the operating system, e.g. --image-family="debian-11" and --image-project=debian-cloud
- See Docu on "Operating system details" for a list of available values
--machine-type
- defines the machine type (i.e. the "specs" like CPUs and memory), e.g. --machine-type=e2-micro
- See Docu on "About machine families" that contains links to the machine type categories with the concrete values, e.g. the General-purpose machine family
etc.

The docu is doing a great job at describing all available configuration options. Luckily, we can make our lives a little easier, configure the VM via UI and then click the "EQUIVALENT COMMAND LINE" button at the end of the page to get a copy-paste-ready gcloud compute instances create command.

Your browser does not support the video tag.

Note that we are using dofroscra-test as the instance name and us-central1-a as the
zone.

gcloud compute instances create dofroscra-test \
    --project=pl-dofroscra-p \
    --zone=us-central1-a \
    --machine-type=e2-micro \
    --network-interface=network-tier=PREMIUM,subnet=default \
    --no-restart-on-failure \
    --maintenance-policy=TERMINATE \
    --provisioning-model=SPOT \
    --instance-termination-action=STOP \
    --service-account=docker-php-tutorial-deployment@pl-dofroscra-p.iam.gserviceaccount.com \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --tags=http-server \
    --create-disk=auto-delete=yes,boot=yes,device-name=dofroscra-test,image=projects/debian-cloud/global/images/debian-11-bullseye-v20220519,mode=rw,size=10,type=projects/pl-dofroscra-p/zones/us-central1-a/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --reservation-affinity=any

Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/zones/us-central1-a/instances/dofroscra-test-1].

NAME            ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
dofroscra-test  us-central1-a  e2-small      true         10.128.0.2   34.122.227.169  RUNNING

Provisioning

For provisioning, we need to install docker and authenticate the root user to pull images from our registry. We've already created an installation script at .infrastructure/scripts/provision.sh and the easiest way to run it on the VM is to transmit it via scp

gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./.infrastructure/scripts/provision.sh dofroscra-test:provision.sh

$ gcloud compute scp --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p ./.infrastructure/scripts/provision.sh dofroscra-test:provision.sh
provision.sh              | 0 kB |   0.7 kB/s | ETA: 00:00:00 | 100%

The previous command transmitted the script in the home directory of the user, and we can now execute it via

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="bash provision.sh"

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="bash provision.sh"
Hit:1 https://download.docker.com/linux/debian bullseye InRelease
Hit:2 http://packages.cloud.google.com/apt cloud-sdk-bullseye InRelease
# ...
The following additional packages will be installed:
  dbus-user-session docker-ce-rootless-extras docker-scan-plugin git git-man
  libcurl3-gnutls liberror-perl libgdbm-compat4 libltdl7 libperl5.32 libslirp0
  patch perl perl-modules-5.32 pigz slirp4netns
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite git-daemon-run | git-daemon-sysvinit
  git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn
  ed diffutils-doc perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl make libtap-harness-archive-perl
The following NEW packages will be installed:
  containerd.io dbus-user-session docker-ce docker-ce-cli
  docker-ce-rootless-extras docker-compose-plugin docker-scan-plugin git
  git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libltdl7 libperl5.32
  libslirp0 patch perl perl-modules-5.32 pigz slirp4netns
0 upgraded, 20 newly installed, 0 to remove and 6 not upgraded.
Need to get 124 MB of archives.
After this operation, 535 MB of additional disk space will be used.
# ...
Setting up docker-ce (5:20.10.16~3-0~debian-bullseye) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up liberror-perl (0.17029-1) ...
Setting up git (1:2.30.2-1) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+deb11u3) ...

Once docker is installed, we can run the authentication of the root user using sudo su root -c via

gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="sudo su root -c 'gcloud auth configure-docker --quiet'"

$ gcloud compute ssh dofroscra-test --zone us-central1-a --tunnel-through-iap --project=pl-dofroscra-p --command="sudo su root -c 'gcloud auth configure-docker --quiet'"
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
gcloud credential helpers already registered correctly.

Deployment

We're almost done - the last step in the process consists of

building the docker image locally using the correct tag gcr.io/pl-dofroscra-p/my-nginx

  docker build -t "gcr.io/pl-dofroscra-p/my-nginx" -f - . <<EOF
  FROM nginx:1.21.5-alpine

  RUN echo "Hello world" >> /usr/share/nginx/html/hello.html

  EOF

  $ docker build -t "gcr.io/pl-dofroscra-p/my-nginx" --no-cache -f - . <<EOF
  FROM nginx:1.21.5-alpine
  RUN echo "Hello world" >> /usr/share/nginx/html/hello.html
  EOF

  #1 [internal] load build definition from Dockerfile
  #1 sha256:21ee68236cc00e4d1638480de32fd6304d796e6a72306082d3164e9164073843

  #...

  #5 [2/2] RUN echo "Hello world" >> /usr/share/nginx/html/hello.html
  #5 sha256:72dee3f3637a6b78ff7e50591e3e9108e2b93eee09443e19890f9cb36b35bdc6
  #5 DONE 0.5s

  #6 exporting to image
  #6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
  #6 exporting layers 0.1s done
  #6 writing image sha256:d0b23a80ebd7311953eeff3818b58007e8747e531e0128eef820f39105f9f1fe done
  #6 naming to gcr.io/pl-dofroscra-p/my-nginx done
  #6 DONE 0.1s

local authentication

  cat ./gcp-service-account-key.json | docker login -u _json_key --password-stdin https://gcr.io

  $ cat ./gcp-service-account-key.json | docker login -u _json_key --password-stdin https://gcr.io
  Login Succeeded

  Logging in with your password grants your terminal complete access to your account.
  For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/

pushing it to the registry

  docker push gcr.io/pl-dofroscra-p/my-nginx


  $ docker push gcr.io/pl-dofroscra-p/my-nginx
  Using default tag: latest
  The push refers to repository [gcr.io/pl-dofroscra-1/my-nginx]
  ad4683501621: Preparing
  # ...
  ad4683501621: Pushed
  latest: digest: sha256:680086b3c77e4b895099c0a5f6e713ff79ba0d78e1e1df1bc2546d6f979126e4 size: 1775

and "deploying it on the VM"

For the last step we will once again use a script that we transmit to the VM: .infrastructure/scripts/deploy.sh

#!/usr/bin/env bash

usage="Usage: deploy.sh image_name"
[ -z "$1" ] &&  echo "No image_name given! $usage" && exit 1

image_name=$1
container_name=nginx

echo "Pulling '${image_name}' from registry"
sudo docker pull "${image_name}"

echo "Starting container"
sudo docker kill "${container_name}"; sudo docker run --name "${container_name}" -p 80:80 --rm -d "${image_name}"

echo "Getting secret GPG_KEY"
gcloud secrets versions access latest --secret=GPG_KEY >> ./secret.gpg
head ./secret.gpg

echo "Getting secret GPG_PASSWORD"
GPG_PASSWORD=$(gcloud secrets versions access latest --secret=GPG_PASSWORD)
echo $GPG_PASSWORD

The script will pull the image on the VM from the registry

$ sudo docker pull 'gcr.io/pl-dofroscra-p/my-nginx'
Using default tag: latest
latest: Pulling from pl-dofroscra-p/my-nginx
# ...
Digest: sha256:680086b3c77e4b895099c0a5f6e713ff79ba0d78e1e1df1bc2546d6f979126e4
Status: Downloaded newer image for gcr.io/pl-dofroscra-p/my-nginx:latest
gcr.io/pl-dofroscra-p/my-nginx:latest

start a container with the image

$ docker kill "${container_name}"; sudo docker run --name "${container_name}" -p 80:80 --rm -d "${image_name}"
Error response from daemon: Cannot kill container: nginx: No such container: nginx
8ac0cc055041e18d3ce244ded49c82be985e580c2c9f316c6d6ef7c7bf3bc0b5

and finally retrieve the secrets (just to demonstrate that it works)

$ gcloud secrets versions access latest --secret=GPG_KEY >> ./secret.gpg
$ head ./secret.gpg
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQPGBGKA1psBCACq5zYDT587CVZEIWXbUplfAGQZOQJALmzErYpTp0jt+rp4vJhR
U5xahy3pqCq81Cnny5YME50ybB3pW/WcHxWLBDo+he8PKeLbp6wFFjJns+3u4opH
9gFMElyHpzTGiDQYfx/CgY2hKz7GSqpjmnOaKxYvGv0EsbZczyHY1WIN/YFzb0tI
tY7J4zTSH05I+aazRdHyn28QcCRcIT9+4q+5Vk8gz8mmgoqVpyeNgQcqJjcd03iP
WUZd1vZCumOvdG5PZNlc/wPFhqLDmYyLmJ7pt5bWIgty9BjYK8Z2NOdUaekqVEJ+
r29HbzwgFLLE2gd52f07h2y2YgMdWdz4FDxVABEBAAH+BwMC9veBYT2oigXxExLl
7fZKVjw02lEr1NpYd5X1ge9WPU/1qumATJWounzciiETpsYGsbPd9zFRJP4E3JZl
sFSh4p0/kXYTuenYD8wgGkeYyN4lm53IHfqSn2z9JMW5Kz9XEODtKJl8fjcn9Zeb

$ GPG_PASSWORD=$(gcloud secrets versions access latest --secret=GPG_PASSWORD)
$ echo $GPG_PASSWORD
87654321

Bonus: For retrieving the puplic IP address of the Compute Instance VM we can use the gcloud compute instances describe $instanceName command:

gcloud compute instances describe dofroscra-test --zone us-central1-a --project=pl-dofroscra-p --format='get(networkInterfaces[0].accessConfigs[0].natIP)'

$ ip=$(gcloud compute instances describe dofroscra-test --zone us-central1-a --project=pl-dofroscra-p --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
$ echo $ip
35.224.250.208
$ curl -s "http://${ip}/hello.html"
Hello world

Putting it all together

Since none of the previous steps requires manual intervention any longer, I have created a script at .infrastructure/setup-gcp.sh to run everything from Configure gcloud to use the master service account to Provisioning:

#!/usr/bin/env bash

usage="Usage: deploy.sh project_id vm_name"
[ -z "$1" ] &&  echo "No project_id given! $usage" && exit 1
[ -z "$2" ] &&  echo "No vm_name given! $usage" && exit 1

GREEN="\033[0;32m"
NO_COLOR="\033[0m"

project_id=$1
vm_name=$2
vm_zone=us-central1-a
master_service_account_key_location=./gcp-master-service-account-key.json
deployment_service_account_id=deployment
deployment_service_account_key_location=./gcp-service-account-key.json
deployment_service_account_mail="${deployment_service_account_id}@${project_id}.iam.gserviceaccount.com"
gpg_secret_key_location=secret-production-protected.gpg.example
gpg_secret_key_password=87654321

printf "${GREEN}Setting up GCP project for${NO_COLOR}\n"
echo "==="
echo "project_id: ${project_id}"
echo "vm_name:    ${vm_name}"

printf "${GREEN}Activating master service account${NO_COLOR}\n"
gcloud auth activate-service-account --key-file="${master_service_account_key_location}" --project="${project_id}"

printf "${GREEN}Enabling APIs${NO_COLOR}\n"
gcloud services enable \
  containerregistry.googleapis.com \
  secretmanager.googleapis.com \
  compute.googleapis.com \
  iam.googleapis.com \
  storage.googleapis.com \
  cloudresourcemanager.googleapis.com

printf "${GREEN}Creating deployment service account with id '${deployment_service_account_id}'${NO_COLOR}\n"
gcloud iam service-accounts create "${deployment_service_account_id}" \
  --description="Used for the deployment application" \
  --display-name="Deployment Account"

printf "${GREEN}Creating JSON key file for deployment service account at ${deployment_service_account_key_location}${NO_COLOR}\n"
gcloud iam service-accounts keys create "${deployment_service_account_key_location}" \
  --iam-account="${deployment_service_account_mail}"

printf "${GREEN}Adding roles for service account${NO_COLOR}\n"  
roles="storage.admin secretmanager.admin compute.admin iam.serviceAccountUser iap.tunnelResourceAccessor"

for role in $roles; do
  gcloud projects add-iam-policy-binding "${project_id}" --member=serviceAccount:"${deployment_service_account_mail}" "--role=roles/${role}"
done;

printf "${GREEN}Creating secrets${NO_COLOR}\n"
gcloud secrets create GPG_KEY
echo gcloud secrets versions add GPG_KEY --data-file="${gpg_secret_key_location}"

gcloud secrets create GPG_PASSWORD
echo -n "${gpg_secret_key_password}" | gcloud secrets versions add GPG_PASSWORD --data-file=-

printf "${GREEN}Creating firewall rule to allow HTTP traffic${NO_COLOR}\n"
gcloud compute firewall-rules create default-allow-http --allow tcp:80 --target-tags=http-server

printf "${GREEN}Creating a Compute Instance VM${NO_COLOR}\n"
gcloud compute instances create "${vm_name}" \
    --project="${project_id}" \
    --zone="${vm_zone}" \
    --machine-type=e2-micro \
    --network-interface=network-tier=PREMIUM,subnet=default \
    --no-restart-on-failure \
    --maintenance-policy=TERMINATE \
    --provisioning-model=SPOT \
    --instance-termination-action=STOP \
    --service-account="${deployment_service_account_mail}" \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --tags=http-server \
    --create-disk=auto-delete=yes,boot=yes,device-name="${vm_name}",image=projects/debian-cloud/global/images/debian-11-bullseye-v20220519,mode=rw,size=10,type=projects/"${project_id}"/zones/"${vm_zone}"/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --reservation-affinity=any

printf "${GREEN}Activating deployment service account${NO_COLOR}\n"
gcloud auth activate-service-account --key-file="${deployment_service_account_key_location}" --project="${project_id}"

printf "${GREEN}Transferring provisioning script${NO_COLOR}\n"
echo "Waiting 60s for the instance to be fully ready to receive IAP connections"
sleep 60
gcloud compute scp --zone ${vm_zone} --tunnel-through-iap --project=${project_id} ./.infrastructure/scripts/provision.sh ${vm_name}:provision.sh

printf "${GREEN}Executing provisioning script${NO_COLOR}\n"
gcloud compute ssh ${vm_name} --zone ${vm_zone} --tunnel-through-iap --project=${project_id} --command="bash provision.sh"

printf "${GREEN}Authenticating docker via gcloud in the VM${NO_COLOR}\n"
gcloud compute ssh ${vm_name} --zone ${vm_zone} --tunnel-through-iap --project=${project_id} --command="sudo su root -c 'gcloud auth configure-docker --quiet'"

printf "\n\n${GREEN}Provisioning done!${NO_COLOR}\n"

$ bash .infrastructure/setup-gcp.sh pl-dofroscra-p dofroscra-test
Setting up GCP project for
===
project_id: pl-dofroscra-p
vm_name:    dofroscra-test
Activating master service account
Activated service account credentials for: [master@pl-dofroscra-p.iam.gserviceaccount.com]
Enabling APIs
Operation "operations/acf.p2-305055072470-625a52a9-96a7-4e2f-831e-a76491c8882c" finished successfully.
Creating deployment service account with id 'deployment'
Created service account [deployment].
Creating JSON key file for deployment service account at ./gcp-service-account-key.json
created key [fa74d605f5891a6f875a2663b6beeea425730b5b] of type [json] as [./gcp-service-account-key.json] for [deployment@pl-dofroscra-p.iam.gserviceaccount.com]
Adding roles for service account
Updated IAM policy for project [pl-dofroscra-p].
bindings:
# ...
- members:
  - serviceAccount:deployment@pl-dofroscra-p.iam.gserviceaccount.com
  role: roles/storage.admin
etag: BwXgw51MPTM=
version: 1
Creating secrets
Created secret [GPG_KEY].
Created version [1] of the secret [GPG_KEY].
Created secret [GPG_PASSWORD].
Created version [1] of the secret [GPG_PASSWORD].
Creating firewall rule to allow HTTP traffic
Creating firewall...
..Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/global/firewalls/default-allow-http].
done.
Creating a Compute Instance VM
Created [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p/zones/us-central1-a/instances/dofroscra-test].
NAME        ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
dofroscra-test  us-central1-a  e2-micro      true         10.128.0.2   34.70.213.101  RUNNING
Activating deployment service account
Activated service account credentials for: [deployment@pl-dofroscra-p.iam.gserviceaccount.com]
Transferring provisioning script
Updating project ssh metadata...
..........................................................................Updated [https://www.googleapis.com/compute/v1/projects/pl-dofroscra-p].
.done.
Waiting for SSH key to propagate.
# ...
provision.sh              | 0 kB |   0.7 kB/s | ETA: 00:00:00 | 100%)
Executing provisioning script
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
# ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+deb11u3) ...
Authenticating docker via gcloud in the VM
# ...
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.


Provisioning done!

In addition, I also created a script for the Deployment at .infrastructure/deploy.sh:

#!/usr/bin/env bash

usage="Usage: deploy.sh project_id vm_name"
[ -z "$1" ] &&  echo "No project_id given! $usage" && exit 1
[ -z "$2" ] &&  echo "No vm_name given! $usage" && exit 1

GREEN="\033[0;32m"
NO_COLOR="\033[0m"

project_id=$1
vm_name=$2
vm_zone=us-central1-a
image_name="gcr.io/${project_id}/nginx:latest"  
container_name=nginx
deployment_service_account_key_location=./gcp-service-account-key.json

printf "${GREEN}Building nginx docker image with name '${image_name}'${NO_COLOR}\n"
docker build -t "${image_name}" -f - . <<EOF
FROM nginx:1.21.5-alpine

RUN echo "Hello world" >> /usr/share/nginx/html/hello.html

EOF

printf "${GREEN}Authenticating docker${NO_COLOR}\n"
cat "${deployment_service_account_key_location}" | docker login -u _json_key --password-stdin https://gcr.io

printf "${GREEN}Pushing '${image_name}' to registry${NO_COLOR}\n"
docker push "${image_name}"

printf "${GREEN}Transferring deployment script${NO_COLOR}\n"
gcloud compute scp --zone ${vm_zone} --tunnel-through-iap --project=${project_id} ./.infrastructure/scripts/deploy.sh ${vm_name}:deploy.sh

printf "${GREEN}Executing deployment script${NO_COLOR}\n"
gcloud compute ssh ${vm_name} --zone ${vm_zone} --tunnel-through-iap --project=${project_id} --command="bash deploy.sh '${image_name}'"

printf "${GREEN}Retrieving external IP of the VM${NO_COLOR}\n"
ip_address=$(gcloud compute instances describe ${vm_name} --zone ${vm_zone} --project=${project_id} --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
printf "http://${ip_address}/hello.html\n"

printf "\n\n${GREEN}Deployment done!${NO_COLOR}\n"

$ bash .infrastructure/deploy.sh pl-dofroscra-p dofroscra-test
Building nginx docker image with name 'gcr.io/pl-dofroscra-p/nginx:latest'
#...
#7 writing image sha256:65ff457111599e3f4dd439138b052cff74f10e3332c593178d8288aebb88bb1c done
#7 naming to gcr.io/pl-dofroscra-p/nginx:latest done
#7 DONE 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Authenticating docker
Login Succeeded

Logging in with your password grants your terminal complete access to your account.
For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/
Pushing 'gcr.io/pl-dofroscra-p/nginx:latest' to registry
The push refers to repository [gcr.io/pl-dofroscra-p/nginx]
ad4683501621: Preparing
# ...
1c9c1e42aafa: Pushed
latest: digest: sha256:680086b3c77e4b895099c0a5f6e713ff79ba0d78e1e1df1bc2546d6f979126e4 size: 1775
Transferring deployment script
deploy.sh                 | 0 kB |   0.6 kB/s | ETA: 00:00:00 | 100%
Executing deployment script
Pulling 'gcr.io/pl-dofroscra-p/nginx:latest' from registry
latest: Pulling from pl-dofroscra-p/nginx
Digest: sha256:f17a9092051c389abf76d254e4d564dbd7a814eb21e3cc47b667db301aa9b497
# ...
gcr.io/pl-dofroscra-p/nginx:latest
Starting container
nginx
58c0a34ca44c9bec97c991bdc69d2353ed75f4214f221e444da36e195a215c75
Getting secret GPG_KEY
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQPGBGKA1psBCACq5zYDT587CVZEIWXbUplfAGQZOQJALmzErYpTp0jt+rp4vJhR
U5xahy3pqCq81Cnny5YME50ybB3pW/WcHxWLBDo+he8PKeLbp6wFFjJns+3u4opH
9gFMElyHpzTGiDQYfx/CgY2hKz7GSqpjmnOaKxYvGv0EsbZczyHY1WIN/YFzb0tI
tY7J4zTSH05I+aazRdHyn28QcCRcIT9+4q+5Vk8gz8mmgoqVpyeNgQcqJjcd03iP
WUZd1vZCumOvdG5PZNlc/wPFhqLDmYyLmJ7pt5bWIgty9BjYK8Z2NOdUaekqVEJ+
r29HbzwgFLLE2gd52f07h2y2YgMdWdz4FDxVABEBAAH+BwMC9veBYT2oigXxExLl
7fZKVjw02lEr1NpYd5X1ge9WPU/1qumATJWounzciiETpsYGsbPd9zFRJP4E3JZl
sFSh4p0/kXYTuenYD8wgGkeYyN4lm53IHfqSn2z9JMW5Kz9XEODtKJl8fjcn9Zeb
Getting secret GPG_PASSWORD
87654321
Retrieving external IP of the VM
http://34.70.213.101/hello.html


Deployment done!

Cleanup

The easiest way to "cleanup everything" that might create any costs, e.g.

the docker images stored on Cloud Storage
the secrets in the Secret Manager
the VM itself

is to delete the whole project, e.g. via the Settings UI

Your browser does not support the video tag.

FYI: GCP projects have 30-day-grace-period during that you can "revert" the deletion.

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now be familiar enough with GCP to create a Compute Instance VM and configure it to run dockerized applications on it.

In the next part of this tutorial, we will deploy our dockerized PHP application "to production" on GCP via docker compose.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

CI Pipelines for dockerized PHP Apps with Github & Gitlab [Tutorial Part 7]

Pascal Landau — Mon, 17 Apr 2023 07:17:19 +0000

How to setup CI (Continuous Integration) pipelines for dockerized PHP applications with Github Actions and Gitlab Pipelines

This article appeared first on https://www.pascallandau.com/ at CI Pipelines for dockerized PHP Apps with Github & Gitlab [Tutorial Part 7]

In the seventh part of this tutorial series on developing PHP on Docker we will setup a CI (Continuous Integration) pipeline to run code quality tools and tests on Github Actions and Gitlab Pipelines.

All code samples are publicly available in my Docker PHP Tutorial repository on Github. You find the branch for this tutorial at part-7-ci-pipeline-docker-php-gitlab-github.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Use git-secret to encrypt secrets in the repository and the following one is A primer on GCP Compute Instance VMs for dockerized Apps.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
- Recommended reading
- Approach
- Try it yourself
CI setup
- General CI notes
  - Initialize make for CI
  - wait-for-service.sh
- Setup for a "local" CI run
  - Run details
  - Execution example
- Setup for Github Actions
  - The Workflow file
- Setup for Gitlab Pipelines
  - The .gitlab-ci.yml pipeline file
- Performance
  - The caching problem on CI
Docker changes
- Compose file updates
  - docker-compose.local.yml
  - docker-compose.ci.yml
  - Adding a health check for mysql
- Build target: ci
  - Build stage ci in the php-base image
    - Use the whole codebase as build context
    - Build the dependencies
    - Create the final image
  - Build stage ci in the application image
- .dockerignore
Makefile changes
- Initialize the shared variables
- ENV based docker compose config
Codebase changes
- Add a test for encrypted files
- Add a password-protected secret gpg key
- Create a JUnit report from PhpUnit
Wrapping up

Introduction

CI is short for Continuous Integration and to me mostly means running the code quality tools and tests of a codebase in an isolated environment (preferably automatically). This is particularly important when working in a team, because the CI system acts as the final gatekeeper before features or bugfixes are merged into the main branch.

I initially learned about CI systems when I stubbed my toes into the open source water. Back in the day I used Travis CI for my own projects and replaced it with Github Actions at some point. At ABOUT YOU we started out with a self-hosted Jenkins server and then moved on to Gitlab CI as a fully managed solution (though we use custom runners).

Approach

In this tutorial I'm going to explain how to make our existing docker setup work with Github Actions and Gitlab CI/CD Pipelines. As I'm a big fan of a "progressive enhancement" approach, we will ensure that all necessary steps can be performed locally through make. This has the additional benefit of keeping a single source of truth (the Makefile) which will come in handy when we set up the CI system on two different providers (Github and Gitlab).

The general process will look very similar to the one for local development:

build the docker setup
start the docker setup
run the qa tools
run the tests

You can see the final results in the CI setup section, including the concrete yml files and links to the repositories, see

Setup for a "local" CI run
Setup for Github Actions
Setup for Gitlab Pipelines

On a code level, we will treat CI as an environment, configured through the env variable ENV. So far we only used ENV=local and we will extend that to also use ENV=ci. The necessary changes are explained after the concrete CI setup instructions in the sections

Docker changes
Makefile changes
Codebase changes

Try it yourself

To get a feeling for what's going on, you can start by executing the local CI run:

checkout branch part-7-ci-pipeline-docker-php-gitlab-github
initialize make
run the .local-ci.sh script

This should give you a similar output as presented in the Execution example.

git checkout part-7-ci-pipeline-docker-php-gitlab-github

# Initialize make
make make-init

# Execute the local CI run
bash .local-ci.sh

CI setup

General CI notes

Initialize `make` for CI

As a very first step we need to "configure" the codebase to operate for the ci environment. This is done through the make-init target as explained later in more detail in the Makefile changes section via

make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"

$ make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"
Created a local .make/.env file

ENV=ci ensures that we

use the correct docker compose config files
use the ci build target

TAG=latest is just a simplification for now because we don't do anything with the images yet. In an upcoming tutorial we will push them to a container registry for later usage in production deployments and then set the TAG to something more meaningful (like the build number).

EXECUTE_IN_CONTAINER=true forces every make command that uses a RUN_IN_*_CONTAINER setup to run in a container. This is important, because the Gitlab runner will actually run in a docker container itself. However, this would cause any affected target to omit the $(DOCKER_COMPOSER) exec prefix.

GPG_PASSWORD=12345678 is the password for the secret gpg key as mentioned in Add a password-protected secret gpg key.

wait-for-service.sh

I'll explain the "container is up and running but the underlying service is not" problem for the mysql service and how we can solve it with a health check later in this article at Adding a health check for mysql. On purpose, we don't want docker compose to take care of the waiting because we can make "better use of the waiting time" and will instead implement it ourselves with a simple bash script located at .docker/scripts/wait-for-service.sh:

#!/bin/bash

name=$1
max=$2
interval=$3

[ -z "$1" ] && echo "Usage example: bash wait-for-service.sh mysql 5 1"
[ -z "$2" ] && max=30
[ -z "$3" ] && interval=1

echo "Waiting for service '$name' to become healthy, checking every $interval second(s) for max. $max times"

while true; do 
  ((i++))
  echo "[$i/$max] ..."; 
  status=$(docker inspect --format "{{json .State.Health.Status }}" "$(docker ps --filter name="$name" -q)")
  if echo "$status" | grep -q '"healthy"'; then 
   echo "SUCCESS";
   break
  fi
  if [ $i == $max ]; then 
    echo "FAIL"; 
    exit 1
  fi 
  sleep $interval; 
done

This script waits for a docker $service to become "healthy" by checking the .State.Health.Status info of the docker inspect command.

CAUTION: The script uses $(docker ps --filter name="$name" -q) to determine the id of the container, i.e. it will "match" all running containers against the $name - this would fail if there is more than one matching container! I.e. you must ensure that $name is specific enough to identify one single container uniquely.

The script will check up to $max times in a interval of $interval seconds. See these answers on the "How do I write a retry logic in script to keep retrying to run it up to 5 times?" question for the implementation of the retry logic. To check the health of the mysql service for 5 times with 1 seconds between each try, it can be called via

bash wait-for-service.sh mysql 5 1

Output

$ bash wait-for-service.sh mysql 5 1
Waiting for service 'mysql' to become healthy, checking every 1 second(s) for max. 5 times
[1/5] ...
[2/5] ...
[3/5] ...
[4/5] ...
[5/5] ...
FAIL

# OR

$ bash wait-for-service.sh mysql 5 1
Waiting for service 'mysql' to become healthy, checking every 1 second(s) for max. 5 times
[1/5] ...
[2/5] ...
SUCCESS

The problem of "container dependencies" isn't new and there are already some existing solutions out there, e.g.

But unfortunately all of them operate by checking the availability of a host:port combination and in the case of mysql that didn't help, because the container was up, the port was reachable but the mysql service in the container was not.

Setup for a "local" CI run

As mentioned under Approach, we want to be able to perform all necessary steps locally and I created a corresponding script at .local-ci.sh:

#!/bin/bash
# fail on any error 
# @see https://stackoverflow.com/a/3474556/413531
set -e

make docker-down ENV=ci || true

start_total=$(date +%s)

# STORE GPG KEY
cp secret-protected.gpg.example secret.gpg

# DEBUG
docker version
docker compose version
cat /etc/*-release || true

# SETUP DOCKER
make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"
start_docker_build=$(date +%s)
make docker-build
end_docker_build=$(date +%s)
mkdir -p .build && chmod 777 .build

# START DOCKER
start_docker_up=$(date +%s)
make docker-up
end_docker_up=$(date +%s)
make gpg-init
make secret-decrypt-with-password

# QA
start_qa=$(date +%s)
make qa || FAILED=true
end_qa=$(date +%s)

# WAIT FOR CONTAINERS
start_wait_for_containers=$(date +%s)
bash .docker/scripts/wait-for-service.sh mysql 30 1
end_wait_for_containers=$(date +%s)

# TEST
start_test=$(date +%s)
make test || FAILED=true
end_test=$(date +%s)

end_total=$(date +%s)

# RUNTIMES
echo "Build docker:        " `expr $end_docker_build - $start_docker_build`
echo "Start docker:        " `expr $end_docker_up - $start_docker_up  `
echo "QA:                  " `expr $end_qa - $start_qa`
echo "Wait for containers: " `expr $end_wait_for_containers - $start_wait_for_containers`
echo "Tests:               " `expr $end_test - $start_test`
echo "---------------------"
echo "Total:               " `expr $end_total - $start_total`

# CLEANUP
# reset the default make variables
make make-init
make docker-down ENV=ci || true

# EVALUATE RESULTS
if [ "$FAILED" == "true" ]; then echo "FAILED"; exit 1; fi

echo "SUCCESS"

Run details

as a preparation step, we first ensure that no outdated ci containers are running (this is only necessary locally, because runners on a remote CI system will start "from scratch")

  make docker-down ENV=ci || true

we take some time measurements to understand how long certain parts take via

  start_total=$(date +%s)

to store the current timestamp

we need the secret gpg key in order to decrypt the secrets and simply copy the password-protected example key (in the actual CI systems the key will be configured as a secret value that is injected in the run)

  # STORE GPG KEY
  cp secret-protected.gpg.example secret.gpg

I like printing some debugging info in order to understand which exact circumstances we're dealing with (tbh, this is mostly relevant when setting the CI system up or making modifications to it)

  # DEBUG
  docker version
  docker compose version
  cat /etc/*-release || true

for the docker setup, we start with initializing the environment for ci

  # SETUP DOCKER
  make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"

then build the docker setup

  make docker-build

and finally add a .build/ directory to
collect the build artifacts

  mkdir -p .build && chmod 777 .build

then, the docker setup is started

  # START DOCKER
  make docker-up

and gpg is initialized so that
the secrets can be decrypted

  make gpg-init
  make secret-decrypt-with-password

We don't need to pass a GPG_PASSWORD to secret-decrypt-with-password because we have set
it up in the previous step as a default value via make-init

once the application container is running, the qa tools are run by invoking the qa make target

  # QA
  make qa || FAILED=true

The || FAILED=true part makes sure that the script will not be terminated if the checks fail.
Instead, the fact that a failure happened is "recorded" in the FAILED variable so that we
can evaluate it at the end. We don't want the script to stop here because we want the
following steps to be executed as well (e.g. the tests).

to mitigate the "mysql is not ready" problem, we will now apply the wait-for-service.sh script

  # WAIT FOR CONTAINERS
  bash .docker/scripts/wait-for-service.sh mysql 30 1

once mysql is ready, we can execute the tests via the test make target and apply the same || FAILED=true workaround as for the qa tools

  # TEST
  make test || FAILED=true

finally, all the timers are printed

  # RUNTIMES
  echo "Build docker:        " `expr $end_docker_build - $start_docker_build`
  echo "Start docker:        " `expr $end_docker_up - $start_docker_up  `
  echo "QA:                  " `expr $end_qa - $start_qa`
  echo "Wait for containers: " `expr $end_wait_for_containers - $start_wait_for_containers`
  echo "Tests:               " `expr $end_test - $start_test`
  echo "---------------------"
  echo "Total:               " `expr $end_total - $start_total`

we clean up the resources (this is only necessary when running locally, because the runner of a CI system would be shut down anyway)

  # CLEANUP
  make make-init
  make docker-down ENV=ci || true

and finally evaluate if any error occurred when running the qa tools or the tests

  # EVALUATE RESULTS
  if [ "$FAILED" == "true" ]; then echo "FAILED"; exit 1; fi

  echo "SUCCESS"

Execution example

Executing the script via

bash .local-ci.sh

yields the following (shortened) output:

$ bash .local-ci.sh
Container dofroscra_ci-redis-1  Stopping
# Stopping all other `ci` containers ...
# ...

Client:
 Cloud integration: v1.0.22
 Version:           20.10.13
# Print more debugging info ...
# ...

Created a local .make/.env file
ENV=ci TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_ci --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose-php-base.yml build php-base
#1 [internal] load build definition from Dockerfile
# Output from building the docker containers 
# ...

ENV=ci TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_ci --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.local.ci.yml -f ./.docker/docker-compose/docker-compose.ci.yml up -d
Network dofroscra_ci_network  Creating
# Starting all `ci` containers ...
# ...

"C:/Program Files/Git/mingw64/bin/make" -s gpg-import GPG_KEY_FILES="secret.gpg"
gpg: directory '/home/application/.gnupg' created
gpg: keybox '/home/application/.gnupg/pubring.kbx' created
gpg: /home/application/.gnupg/trustdb.gpg: trustdb created
gpg: key D7A860BBB91B60C7: public key "Alice Doe protected <alice.protected@example.com>" imported
# Output of importing the secret and public gpg keys
# ...

"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f -p 12345678"
git-secret: done. 1 of 1 files are revealed.
"C:/Program Files/Git/mingw64/bin/make" -j 8 -k --no-print-directory --output-sync=target qa-exec NO_PROGRESS=true
phplint                             done   took 4s
phpcs                               done   took 4s
phpstan                             done   took 8s
composer-require-checker            done   took 8s
Waiting for service 'mysql' to become healthy, checking every 1 second(s) for max. 30 times
[1/30] ...
SUCCESS
PHPUnit 9.5.19 #StandWithUkraine

........                                                            8 / 8 (100%)

Time: 00:03.077, Memory: 28.00 MB

OK (8 tests, 15 assertions)
Build docker:         12
Start docker:         2
QA:                   9
Wait for containers:  3
Tests:                5
---------------------
Total:                46
Created a local .make/.env file

Container dofroscra_ci-application-1  Stopping
Container dofroscra_ci-mysql-1  Stopping
# Stopping all other `ci` containers ...
# ...

SUCCESS

Setup for Github Actions

If you are completely new to Github Actions, I recommend to start with the official Quickstart Guide for GitHub Actions and the Understanding GitHub Actions article. In short:

Github Actions are based on so called Workflows
- Workflows are yaml files that live in the special .github/workflows directory in the repository
a Workflow can contain multiple Jobs
each Job consists of a series of Steps
each Step needs a run: element that represents a command that is executed by a new shell
- multi-line commands that should use the same shell are written as
```
  - run : |
        echo "line 1"
        echo "line 2"
```
See also difference between "run |" and multiple runs in github actions

The Workflow file

Github Actions are triggered automatically based on the files in the .github/workflows directory. I have added the file .github/workflows/ci.yml with the following content:

name: CI build and test

on:
  # automatically run for pull request and for pushes to branch "part-7-ci-pipeline-docker-php-gitlab-github"
  # @see https://stackoverflow.com/a/58142412/413531
  push:
    branches:
      - part-7-ci-pipeline-docker-php-gitlab-github
  pull_request: {}
  # enable to trigger the action manually
  # @see https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/
  # CAUTION: there is a known bug that makes the "button to trigger the run" not show up
  # @see https://github.community/t/workflow-dispatch-workflow-not-showing-in-actions-tab/130088/29
  workflow_dispatch: {}

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v1

      - name: start timer
        run: |
          echo "START_TOTAL=$(date +%s)" > $GITHUB_ENV

      - name: STORE GPG KEY
        run: |
          # Note: make sure to wrap the secret in double quotes (")
          echo "${{ secrets.GPG_KEY }}" > ./secret.gpg

      - name: SETUP TOOLS
        run : |
          DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
          # install docker compose
          # @see https://docs.docker.com/compose/cli-command/#install-on-linux
          # @see https://github.com/docker/compose/issues/8630#issuecomment-1073166114
          mkdir -p $DOCKER_CONFIG/cli-plugins 
          curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-$(uname -m) -o $DOCKER_CONFIG/cli-plugins/docker-compose
          chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

      - name: DEBUG
        run: |
          docker compose version
          docker --version
          cat /etc/*-release

      - name: SETUP DOCKER
        run: |
          make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=${{ secrets.GPG_PASSWORD }}"
          make docker-build
          mkdir .build && chmod 777 .build

      - name: START DOCKER
        run: |
          make docker-up
          make gpg-init
          make secret-decrypt-with-password

      - name: QA
        run: |
          # Run the tests and qa tools but only store the error instead of failing immediately
          # @see https://stackoverflow.com/a/59200738/413531
          make qa || echo "FAILED=qa" >> $GITHUB_ENV

      - name: WAIT FOR CONTAINERS
        run: |
          # We need to wait until mysql is available.
          bash .docker/scripts/wait-for-service.sh mysql 30 1 

      - name: TEST
        run: |
          make test || echo "FAILED=test $FAILED" >> $GITHUB_ENV

      - name: RUNTIMES
        run: |
          echo `expr $(date +%s) - $START_TOTAL`

      - name: EVALUATE
        run: |
          # Check if $FAILED is NOT empty
          if [ ! -z "$FAILED" ]; then echo "Failed at $FAILED" && exit 1; fi

      - name: upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: build-artifacts
          path: ./.build

The steps are essentially the same as explained before at Run details for the local run. Some additional notes:

I want the Action to be triggered automatically only when I push to branch part-7-ci-pipeline-docker-php-gitlab-github OR when a pull request is created (via pull_request). In addition, I want to be able to trigger the Action manually on any branch (via workflow_dispatch).

  on:
    push:
      branches:
        - part-7-ci-pipeline-docker-php-gitlab-github
    pull_request: {}
    workflow_dispatch: {}

For a real project, I would let the action only run automatically on long-living branches like
main or develop. The manual trigger is helpful if you just want to test your current work
without putting it up for review. CAUTION: There is a
known issue that "hides" the "Trigger workflow" button to trigger the action manually.

a new shell is started for each run: instruction, thus we must store our timer in the "global" environment variable $GITHUB_ENV

      - name: start timer
      run: |
        echo "START_TOTAL=$(date +%s)" > $GITHUB_ENV

This will be the only timer we use, because the job uses multiple steps that are timed
automatically - so we don't need to take timestamps manually:

the gpg key is configured as an encrypted secret named GPG_KEY and is stored in ./secret.gpg. The value is the content of the secret-protected.gpg.example file

      - name: STORE GPG KEY
        run: |
          echo "${{ secrets.GPG_KEY }}" > ./secret.gpg

Secrets are configured in the Github repository under Settings > Secrets > Actions at

  https://github.com/$user/$repository/settings/secrets/actions

  e.g.

  https://github.com/paslandau/docker-php-tutorial/settings/secrets/actions

the ubuntu-latest image doesn't contain the docker compose plugin, thus we need to install it manually

      - name: SETUP TOOLS
      run : |
        DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
        mkdir -p $DOCKER_CONFIG/cli-plugins 
        curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-$(uname -m) -o $DOCKER_CONFIG/cli-plugins/docker-compose
        chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

for the make initialization we need the second secret named GPG_PASSWORD - which is configured as 12345678 in our case, see Add a password-protected secret gpg key

      - name: SETUP DOCKER
        run: |
          make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=${{ secrets.GPG_PASSWORD }}"

because the runner will be shutdown after the run, we need to move the build artifacts to a permanent location, using the actions/upload-artifact@v3 action

      - name: upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: build-artifacts
          path: ./.build

You can
download the artifacts in the Run overview UI

Setup for Gitlab Pipelines

If you are completely new to Gitlab Pipelines, I recommend to start with the official Get started with GitLab CI/CD guide. In short:

the core concept of Gitlab Pipelines is the Pipeline
- it is defined in the yaml file .gitlab-ci.yml that lives in the root of the repository
a Pipeline can contain multiple Stages
each Stage consists of a series of Jobs
each Job contains a script section
the script section consists of a series of shell commands

The `.gitlab-ci.yml` pipeline file

Gitlab Pipelines are triggered automatically based on a .gitlab-ci.yml file located at the root of the repository. It has the following content:

stages:
  - build_test

QA and Tests:
  stage: build_test

  rules:
    # automatically run for pull request and for pushes to branch "part-7-ci-pipeline-docker-php-gitlab-github"
    - if: '($CI_PIPELINE_SOURCE == "merge_request_event" || $CI_COMMIT_BRANCH == "part-7-ci-pipeline-docker-php-gitlab-github")'

  # see https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker
  image: docker:20.10.12

  services:
    - name: docker:dind

  script:
    - start_total=$(date +%s)

    ## STORE GPG KEY
    - cp $GPG_KEY_FILE ./secret.gpg

    ## SETUP TOOLS
    - start_install_tools=$(date +%s)
    # "curl" is required to download docker compose
    - apk add --no-cache make bash curl
    # install docker compose
    # @see https://docs.docker.com/compose/cli-command/#install-on-linux
    - mkdir -p ~/.docker/cli-plugins/
    - curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
    - chmod +x ~/.docker/cli-plugins/docker-compose
    - end_install_tools=$(date +%s)

    ## DEBUG
    - docker version
    - docker compose version
    # show linux distro info
    - cat /etc/*-release

    ## SETUP DOCKER
    # Pass default values to the make-init command - otherwise we would have to pass those as arguments to every make call
    - make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=$GPG_PASSWORD"
    - start_docker_build=$(date +%s)
    - make docker-build
    - end_docker_build=$(date +%s)
    - mkdir .build && chmod 777 .build

    ## START DOCKER
    - start_docker_up=$(date +%s)
    - make docker-up
    - end_docker_up=$(date +%s)
    - make gpg-init
    - make secret-decrypt-with-password

    ## QA
    # Run the tests and qa tools but only store the error instead of failing immediately
    # @see https://stackoverflow.com/a/59200738/413531
    - start_qa=$(date +%s)
    - make qa ENV=ci || FAILED=true
    - end_qa=$(date +%s)

    ## WAIT FOR CONTAINERS
    # We need to wait until mysql is available.
    - start_wait_for_containers=$(date +%s)
    - bash .docker/scripts/wait-for-service.sh mysql 30 1
    - end_wait_for_containers=$(date +%s)

    ## TEST
    - start_test=$(date +%s)
    - make test ENV=ci || FAILED=true
    - end_test=$(date +%s)

    - end_total=$(date +%s)

    # RUNTIMES
    - echo "Tools:" `expr $end_install_tools - $start_install_tools`
    - echo "Build docker:" `expr $end_docker_build - $start_docker_build`
    - echo "Start docker:" `expr $end_docker_up - $start_docker_up  `
    - echo "QA:" `expr $end_qa - $start_qa`
    - echo "Wait for containers:" `expr $end_wait_for_containers - $start_wait_for_containers`
    - echo "Tests:" `expr $end_test - $start_test`
    - echo "Total:" `expr $end_total - $start_total`

    # EVALUATE RESULTS
    # Use if-else constructs in Gitlab pipelines
    # @see https://stackoverflow.com/a/55464100/413531
    - if [ "$FAILED" == "true" ]; then exit 1; fi

  # Save the build artifact, e.g. the JUNIT report.xml file, so we can download it later
  # @see https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html
  artifacts:
    when: always
    paths:
      # the quotes are required
      # @see https://stackoverflow.com/questions/38009869/how-to-specify-wildcard-artifacts-subdirectories-in-gitlab-ci-yml#comment101411265_38055730
      - ".build/*"
    expire_in: 1 week

The steps are essentially the same as explained before under Run details for the local run. Some additional notes:

we start by defining the stages of the pipeline - though that's currently just one (build_test)

  stages:
    - build_test

then we define the job QA and Tests and assign it to the build_test stage

  QA and Tests:
    stage: build_test

I want the Pipeline to be triggered automatically only when I push to branch part-7-ci-pipeline-docker-php-gitlab-github OR when a pull request is created Triggering the Pipeline manually on any branch is possible by default.

    rules:
    - if: '($CI_PIPELINE_SOURCE == "merge_request_event" || $CI_COMMIT_BRANCH == "part-7-ci-pipeline-docker-php-gitlab-github")'

since we want to build and run docker images, we need to use a docker base image and activate the docker:dind service. See Use Docker to build Docker images: Use Docker-in-Docker

  image: docker:20.10.12

  services:
    - name: docker:dind

we store the secret gpg key as a secret file (using the "file" type) in the CI/CD variables configuration of the Gitlab repository and move it to ./secret.gpg in order to decrypt the secrets later

    ## STORE GPG KEY
    - cp $GPG_KEY_FILE ./secret.gpg

Secrets can be configured under Settings > CI/CD > Variables at

  https://gitlab.com/$project/$repository/-/settings/ci_cd

  e.g.

  https://gitlab.com/docker-php-tutorial/docker-php-tutorial/-/settings/ci_cd

the docker base image doesn't come with all required tools, thus we need to install the missing ones (make, bash, curl and docker compose)

      ## SETUP TOOLS
      - apk add --no-cache make bash curl
      - mkdir -p ~/.docker/cli-plugins/
      - curl -sSL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
      - chmod +x ~/.docker/cli-plugins/docker-compose

for the initialization of make we use the $GPG_PASSWORD variable that we defined in the CI/CD settings

   ## SETUP DOCKER
    - make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=$GPG_PASSWORD"

Note: I have marked the variable as "masked"
so it won't show up in any logs

finally, we store the job artifacts

  artifacts:
    when: always
    paths:
      - ".build/*"
    expire_in: 1 week

They can be accessed in the Pipeline overview UI

Performance

Performance isn't an issue right now, because the CI runs take only about ~1 min (Github Actions) and ~2 min (Gitlab Pipelines), but that's mostly because we only ship a super minimal application and those times will go up when things get more complex. For the local setup I used all 8 cores of my laptop. The time breakdown is roughly as follows:

Step	Gitlab	Github	local without cache	local with cached images	local with cached images + layers
SETUP TOOLS	1	0	0	0	0
SETUP DOCKER	33	17	39	39	5
START DOCKER	17	11	34	2	1
QA	17	5	10	13	1
WAIT FOR CONTAINERS	5	5	3	2	13
TESTS	3	1	3	6	3
total (excl. runner startup)	78	43	97	70	36
total (incl. runner startup)	139	54	97	70	36

Times taken from

"CI build and test #83" Github Action run
"Pipeline #511355192" Gitlab Pipeline run
"local without cache" via bash .local-ci.sh with no local images at all
"local with cached images" via bash .local-ci.sh with cached images for mysql and redis
"local with cached images + layers" via bash .local-ci.sh with cached images for mysql and redis and a "warm" layer cache for the application image

Optimizing the performance is out of scope for this tutorial, but I'll at least document my current findings.

The caching problem on CI

A good chunk of time is usually spent on building the docker images. We did our best to optimize the process by leveraging the layer cache and using cache mounts (see section Build stage ci in the php-base image). But those steps are futile on CI systems, because the corresponding runners will start "from scratch" for every CI run - i.e. there is no local cache that they could use. In consequence, the full docker setup is also built "from scratch" on every run.

There are ways to mitigate that e.g.

pushing images to a container registry and pulling them before building the images to leverage the layer cache via the cache_from option of docker compose
exporting and importing the images as tar archives via docker save and docker load, storing them either in the built-in cache of Github or Gitlab
- see also the satackey/action-docker-layer-caching@v0.0.11 Github Action and the official actions/cache@v3 Github Action
using the --cache-from and --cache-to options of buildx
- see also the "cache" docu of the docker/build-push-action@v2 Github Action

But: None of that worked for me out-of-the-box :( We will take a closer look in an upcoming tutorial. Some reading material that I found valuable so far:

Docker changes

As a first step we need to decide which containers are required and how to provide the codebase.

Since our goal is running the qa tools and tests, we only need the application php container. The tests also need a database and a queue, i.e. the mysql and redis containers are required as well - whereas nginx, php-fpm and php-worker are not required. We'll handle that through dedicated docker compose configuration files that only contain the necessary services. This is explained in more detail in section Compose file updates.

In our local setup, we have sheen the host system and docker - mainly because we wanted our changes to be reflected immediately in docker.

This isn't necessary for the CI use case. In fact we want our CI images as close as possible to our production images - and those should "contain everything to run independently". I.e. the codebase should live in the image - not on the host system. This will be explained in section Use the whole codebase as build context.

Compose file updates

We will not only have some differences between the CI docker setup and the local docker setup (=different containers), but also in the configuration of the individual services. To accommodate for that, we will use the following docker compose config files in the .docker/docker-compose/ directory:

docker-compose.local.ci.yml:
- holds configuration that is valid for local and ci, trying to keep the config files DRY
docker-compose.ci.yml:
- holds configuration that is only valid for ci
docker-compose.local.yml:
- holds configuration that is only valid for local

When using docker compose we then need to make sure to include only the required files, e.g. for ci:

docker compose -f docker-compose.local.ci.yml -f docker-compose.ci.yml

I'll explain the logic for that later in section ENV based docker compose config. In short:

docker-compose.local.yml

When comparing ci with local, for ci

we don't need to share the codebase with the host system

    application:
      volumes:
      - ${APP_CODE_PATH_HOST?}:${APP_CODE_PATH_CONTAINER?}

we don't need persistent volumes for the redis and mysql data

    mysql:
      volumes:
        - mysql:/var/lib/mysql

    redis:
      volumes:
        - redis:/data

we don't need to share ports with the host system

    application:
      ports:
        - "${APPLICATION_SSH_HOST_PORT:-2222}:22"

    redis:
      ports:
        - "${REDIS_HOST_PORT:-6379}:6379"

we don't need any settings for local dev tools like xdebug or strace

    application:
      environment:
        - PHP_IDE_CONFIG=${PHP_IDE_CONFIG?}
      cap_add:
        - "SYS_PTRACE"
      security_opt:
        - "seccomp=unconfined"
      extra_hosts:
        - host.docker.internal:host-gateway

So all of those config values will only live in the docker-compose.local.yml file.

docker-compose.ci.yml

In fact, there are only two things that ci needs that local doesn't:

a bind mount to share only the secret gpg key from the host with the application container

    application:
      volumes:
        - ${APP_CODE_PATH_HOST?}/secret.gpg:${APP_CODE_PATH_CONTAINER?}/secret.gpg:ro

This
is required to decrypt the secrets:

[...] the private key has to be named secret.gpg and put in the root of the codebase,
so that the import can be simplified with make targets

The secret files themselves are baked into the image, but the key to decrypt them will be
provided only during runtime and

a bind mount to share a .build folder for build artifacts with the application container

    application:
      volumes:
        - ${APP_CODE_PATH_HOST?}/.build:${APP_CODE_PATH_CONTAINER?}/.build

This will be used to collect any files we want to retain from a build (e.g. code coverage
information, log files, etc.)

Adding a health check for `mysql`

When running the tests for the first time on a CI system, I noticed some weird errors related to the database:

1) Tests\Feature\App\Http\Controllers\HomeControllerTest::test___invoke with data set "default" (array(), '    <li><a href="?dispatch=fo...></li>')
PDOException: SQLSTATE[HY000] [2002] Connection refused

As it turned out, the mysql container itself was up and running - but the mysql process within the container was not yet ready to accept connections. Locally, this hasn't been a problem, because we usually would not run the tests "immediately" after starting the containers - but on CI this is the case.

Fortunately, docker compose has us covered here and provides a healtcheck configuration option:

healthcheck declares a check that’s run to determine whether or not containers for this service are "healthy".

Since this healthcheck is also "valid" for local, I defined it in the combined docker-compose.local.ci.yml file:

  mysql:
    healthcheck:
      # Only mark the service as healthy if mysql is ready to accept connections
      # Check every 2 seconds for 30 times, each check has a timeout of 1s
      test: mysqladmin ping -h 127.0.0.1 -u $$MYSQL_USER --password=$$MYSQL_PASSWORD
      timeout: 1s
      retries: 30
      interval: 2s

The script in test was taken from SO: Docker-compose check if mysql connection is ready.

When starting the docker setup, docker ps will now add a health info to the STATUS:

$ make docker-up

$ docker ps
CONTAINER ID   IMAGE                            STATUS                           NAMES
b509eb2f99c0   dofroscra/application-ci:latest  Up 1 seconds                     dofroscra_ci-application-1
503e52fd9e68   mysql:8.0.28                     Up 1 seconds (health: starting)  dofroscra_ci-mysql-1

# a couple of seconds later

$ docker ps
CONTAINER ID   IMAGE                            STATUS                   NAMES
b509eb2f99c0   dofroscra/application-ci:latest  Up 13 seconds            dofroscra_ci-application-1
503e52fd9e68   mysql:8.0.28                     Up 13 seconds (healthy)  dofroscra_ci-mysql-1

Note the (health: starting) and (healthy) infos for the mysql service.

We can also get this info from docker inspect (used by our wait-for-service.sh script) via:

$ docker inspect --format "{{json .State.Health.Status }}" dofroscra_ci-mysql-1
"healthy"

FYI: We could also use the depends_on property with a condition: service_healthy on the application container so that docker compose would only start the container once the mysql service is healthy:

application:
  depends_on:
    mysql: 
      condition: service_healthy

However, this would "block" the make docker-up until mysql is actually up and running. In our case this is not desirable, because we can do "other stuff" in the meantime (namely: run the qa checks, because they don't require a database) and thus save a couple of seconds on each CI run.

Build target: `ci`

We've already introduced build targets in Environments and build targets and how to "choose" them through make with the ENV variable defined in a shared .make/.env file. Short recap:

create a .make/.env file via make make-init that contains the ENV, e.g.

  ENV=ci

the .make/.env file is included in the main Makefile, making the ENV variables available to make
configure a $DOCKER_COMPOSE variable that passes the ENV as an environment variable, i.e. via

  ENV=$(ENV) docker-compose

use the ENV variable in the docker compose configuration file to determine the build.target property. E.g. in .docker/docker-compose/docker-compose-php-base.yml

    php-base:
      build:
        target: ${ENV?}

in the Dockerfile of a service, define the ENV as a build stage. E.g. in .docker/images/php/base/Dockerfile

  FROM base as ci
  # ...

So to enable the new ci environment, we need to modify the Dockerfiles for the php-base and the application image.

Build stage `ci` in the `php-base` image

Use the whole codebase as build context

As mentioned in section Docker changes we want to "bake" the codebase into the ci image of the php-base container.

Thus, we must change the context property in .docker/docker-compose/docker-compose-php-base.yml to not only use the .docker/ directory but instead the whole codebase. I.e. "dont use ../ but ../../":

# File: .docker/docker-compose/docker-compose-php-base.yml

  php-base:
    build:
      # pass the full codebase to docker for building the image
      context: ../../

Build the dependencies

The composer dependencies must be set up in the image as well, so we introduce a new stage stage in .docker/images/php/base/Dockerfile. The most trivial solution would look like this:

copy the whole codebase
run composer install

FROM base as ci

COPY . /codebase

RUN composer install --no-scripts --no-plugins --no-progress -o

However, this approach has some downsides:

if any file in the codebase changes, the COPY . /codebase layer will be invalidated. I.e. docker could not use the layer cache which also means that every layer afterwards cannot use the cache as well. In consequence the composer install would run every time - even when the composer.json file doesn't change.
composer itself uses a cache for storing dependencies locally so it doesn't have to download dependencies that haven't changed. But since we run composer install in Docker, this cache would be "thrown away" every time a build finishes. To mitigate that, we can use --mount=type=cache to define a directory that docker will re-use between builds: > Contents of the cache directories persists between builder invocations without invalidating > the instruction cache.

Keeping those points in mind, we end up with the following instructions:

# File: .docker/images/php/base/Dockerfile
# ...

FROM base as ci

# By only copying the composer files required to run composer install
# the layer will be cached and only invalidated when the composer dependencies are changed
COPY ./composer.json /dependencies/
COPY ./composer.lock /dependencies/

# use a cache mount to cache the composer dependencies
# this is essentially a cache that lives in Docker BuildKit (i.e. has nothing to do with the host system) 
RUN --mount=type=cache,target=/tmp/.composer \
    cd /dependencies && \
    # COMPOSER_HOME=/tmp/.composer sets the home directory of composer that
    # also controls where composer looks for the cache 
    # so we don't have to download dependencies again (if they are cached)
    COMPOSER_HOME=/tmp/.composer composer install --no-scripts --no-plugins --no-progress -o 

# copy the full codebase
COPY . /codebase

RUN mv /dependencies/vendor /codebase/vendor && \
    cd /codebase && \
    # remove files we don't require in the image to keep the image size small
    rm -rf .docker/ && \
    # we need a git repository for git-secret to work (can be an empty one)
    git init

FYI: The COPY . /codebase step doesn't actually copy "everything in the repository", because we have also introduced a .dockerignore file to exclude some files from being included in the build context - see section .dockerignore.

Some notes on the final RUN step:

rm -rf .docker/ doesn't really save "that much" in the current setup - please take it more as an example to remove any files that shouldn't end up in the final image (e.g. "tests in a production image")
the git init part is required because we need to decrypt the secrets later - and git-secret requires a git repository (which can be empty). We can't decrypt the secrets during the build, because we do not want decrypted secret files to end up in the image.

When tested locally, the difference between the trivial solution and the one that makes use of layer caching is ~35 seconds, see the results in the Performance section.

Create the final image

As a final step, we will rename the current stage to codebase and copy the "build artifact" from that stage into our final ci build stage:

FROM base as codebase

# build the composer dependencies and clean up the copied files
# ...

FROM base as ci

COPY --from=codebase --chown=$APP_USER_NAME:$APP_GROUP_NAME /codebase $APP_CODE_PATH

Why are we not just using the previous stage directly as ci?

Because using multistage-builds is a good practice to keep the final layers of an image to a minimum: Everything that "happened" in the previous codebase stage will be "forgotten", i.e. not exported as layers.

That does not only save us some layers, but also allows us to get rid of files like the .docker/ directory. We needed that directory in the build context because some files where required in other parts of the Dockerfile (e.g. the php ini files), so we can't exclude it via .dockerignore. But we can remove it in the codebase stage - so it will NOT be copied over and thus not end up in the final image. If we wouldn't have the codebase stage, the folder would be part of the layer created when COPYing all the files from the build context and removing it via rm -rf .docker/ would have no effect on the image size.

Currently, that doesn't really matter, because the building step is super simple (just a composer install) - but in a growing and more complex codebase you can easily save a couple MB.

To be concrete, the multistage build has 31 layers and the final layer containing the codebase has a size of 65.1MB.

$ docker image history -H dofroscra/application-ci
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
d778c2ee8d5e   17 minutes ago   COPY /codebase /var/www/app # buildkit          65.1MB    buildkit.dockerfile.v0
                                                                                ^^^^^^
<missing>      17 minutes ago   WORKDIR /var/www/app                            0B        buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY /usr/bin/composer /usr/local/bin/compos…   2.36MB    buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/.bashrc /root…   395B      buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/.bashrc /home…   395B      buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   196B      buildkit.dockerfile.v0
<missing>      17 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   378B      buildkit.dockerfile.v0
<missing>      17 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   1.28kB    buildkit.dockerfile.v0
<missing>      17 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   41MB      buildkit.dockerfile.v0
<missing>      18 minutes ago   ADD https://php.hernandev.com/key/php-alpine…   451B      buildkit.dockerfile.v0
<missing>      18 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   62.1MB    buildkit.dockerfile.v0
<missing>      18 minutes ago   ADD https://gitsecret.jfrog.io/artifactory/a…   450B      buildkit.dockerfile.v0
<missing>      18 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   4.74kB    buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV ENV=ci                                      0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV ALPINE_VERSION=3.15                         0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV TARGET_PHP_VERSION=8.1                      0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_CODE_PATH=/var/www/app                  0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_GROUP_NAME=application                  0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_USER_NAME=application                   0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_GROUP_ID=10001                          0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ENV APP_USER_ID=10000                           0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG ENV                                         0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG ALPINE_VERSION                              0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG TARGET_PHP_VERSION                          0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_CODE_PATH                               0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_GROUP_NAME                              0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_USER_NAME                               0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_GROUP_ID                                0B        buildkit.dockerfile.v0
<missing>      18 minutes ago   ARG APP_USER_ID                                 0B        buildkit.dockerfile.v0
<missing>      2 days ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      2 days ago       /bin/sh -c #(nop) ADD file:5d673d25da3a14ce1…   5.57MB

The non-multistage build has 32 layers and the final layer(s) containing the codebase have a combined size of 65.15MB (60.3MB + 4.85MB).

$ docker image history -H dofroscra/application-ci
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
94ba50438c9a   2 minutes ago    RUN /bin/sh -c COMPOSER_HOME=/tmp/.composer …   60.3MB    buildkit.dockerfile.v0
<missing>      2 minutes ago    COPY . /var/www/app # buildkit                  4.85MB    buildkit.dockerfile.v0
                                                                                ^^^^^^
<missing>      31 minutes ago   WORKDIR /var/www/app                            0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY /usr/bin/composer /usr/local/bin/compos…   2.36MB    buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/.bashrc /root…   395B      buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/.bashrc /home…   395B      buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   196B      buildkit.dockerfile.v0
<missing>      31 minutes ago   COPY ./.docker/images/php/base/conf.d/zz-app…   378B      buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   1.28kB    buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   41MB      buildkit.dockerfile.v0
<missing>      31 minutes ago   ADD https://php.hernandev.com/key/php-alpine…   451B      buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   62.1MB    buildkit.dockerfile.v0
<missing>      31 minutes ago   ADD https://gitsecret.jfrog.io/artifactory/a…   450B      buildkit.dockerfile.v0
<missing>      31 minutes ago   RUN |8 APP_USER_ID=10000 APP_GROUP_ID=10001 …   4.74kB    buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV ENV=ci                                      0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV ALPINE_VERSION=3.15                         0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV TARGET_PHP_VERSION=8.1                      0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_CODE_PATH=/var/www/app                  0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_GROUP_NAME=application                  0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_USER_NAME=application                   0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_GROUP_ID=10001                          0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ENV APP_USER_ID=10000                           0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG ENV                                         0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG ALPINE_VERSION                              0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG TARGET_PHP_VERSION                          0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_CODE_PATH                               0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_GROUP_NAME                              0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_USER_NAME                               0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_GROUP_ID                                0B        buildkit.dockerfile.v0
<missing>      31 minutes ago   ARG APP_USER_ID                                 0B        buildkit.dockerfile.v0
<missing>      2 days ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      2 days ago       /bin/sh -c #(nop) ADD file:5d673d25da3a14ce1…   5.57MB

Again: It is expected that the differences aren't big, because the only size savings come from the .docker/ directory with a size of ~70kb.

$ du -hd 0 .docker
73K     .docker

Finally, we are also using the --chown option of the RUN instruction to ensure that the files have the correct permissions.

Build stage `ci` in the `application` image

There is actually "nothing" to be done here. We don't need SSH any longer because it is only required for the SSH Configuration of PhpStorm. So the build stage is simply "empty":

ARG BASE_IMAGE
FROM ${BASE_IMAGE} as base

FROM base as ci

FROM base as local
# ...

Though there is one thing to keep in mind: In the local image we used sshd as the entrypoint, i.e. we had a long running process that would keep the container running. To keep the ci application container running, we must

start it via the -d flag of docker compose (already done in the make docker-up target)

  .PHONY: docker-up
  docker-up: validate-docker-variables
      $(DOCKER_COMPOSE) up -d $(DOCKER_SERVICE_NAME)

allocate a tty via tty: true in the docker-compose.local.ci.yml file

    application:
      tty: true

.dockerignore

The .dockerignore file is located in the root of the repository and ensures that certain files are kept out of the Docker build context. This will

speed up the build (because less files need to be transmitted to the docker daemon)
keep images smaller (because irrelevant files are kept out of the image)

The syntax is quite similar to the .gitignore file - in fact I've found it to be quite often the case that the contents of the .gitignore file are a subset of the .dockerignore file. This makes kinda sense, because you typically wouldn't want files that are excluded from the repository to end up in a docker image (e.g. unencrypted secret files). This has also been noticed by others, see e.g.

but to my knowledge there is currently (2022-04-24) no way to "keep the two files in sync".

CAUTION: The behavior between the two files is NOT identical! The documentation says

Matching is done using Go’s filepath.Match rules. A preprocessing step removes leading and
trailing whitespace and eliminates . and .. elements using Go’s filepath.Clean. Lines that are blank after preprocessing are ignored.

Beyond Go’s filepath.Match rules, Docker also supports a special wildcard string ** that
matches any number of directories (including zero). For example, **/*.go will exclude all
files that end with .go that are found in all directories, including the root of the build context.

Lines starting with ! (exclamation mark) can be used to make exceptions to exclusions.

Please note the part regarding **\*.go: In .gitignore it would be sufficient to write .go to match any file that contains .go, regardless of the directory. In .dockerignore you must specify it as **/*.go!

In our case, the content of the .dockerignore file looks like this:

# gitignore
!.env.example
**/*.env .idea .phpunit.result.cache vendor/
secret.gpg .gitsecret/keys/random_seed .gitsecret/keys/pubring.kbx~
!*.secret passwords.txt .build

# additionally ignored files .git ```



<!-- generated -->
<a id='makefile-changes'> </a>
<!-- /generated -->

## Makefile changes

<!-- generated -->
<a id='initialize-the-shared-variables'> </a>
<!-- /generated -->

### Initialize the shared variables

We have introduced the concept of [shared variables via `.make/.env`](https://www.pascallandau.com/blog/docker-from-scratch-for-php-applications-in-2022/#shared-variables-make-env)
previously. It allows us to **define variables in one place** (=single source 
of truth) that are then used as "defaults" so we **don't have to define them explicitly** when 
invoking certain `make` targets (like `make docker-build`). We'll make use of this concept by 
setting the environment to `ci`via`ENV=ci` and thus making sure that all docker commands use 
`ci` "automatically" as well.

[![Initialize make to run docker commands with ENV=ci](https://www.pascallandau.com/img/ci-pipeline-docker-php-gitlab-github/make-init-ci-docker-commands.PNG)](https://www.pascallandau.com/img/ci-pipeline-docker-php-gitlab-github/make-init-ci-docker-commands.PNG)

In addition, I made a small modification by **introducing a second file at `.make/variables.env`** 
that is also included in the main `Makefile` and **holds the "default" shared variables**. Those 
are neither "secret" nor are they likely to be changed for environment adjustments. The file 
is NOT ignored by `.gitignore` and is basically just the previous `.make/.env.example` file without 
the environment specific variables:



```text
# File .make/variables.env

DOCKER_REGISTRY=docker.io
DOCKER_NAMESPACE=dofroscra
APP_USER_NAME=application
APP_GROUP_NAME=application

The .make/.env file is still .gitignored and can be initialized with the make-init
target using the ENVS variable:

make make-init ENVS="ENV=ci SOME_OTHER_DEFAULT_VARIABLE=foo"

which would create a .make/.env file with the content

ENV=ci SOME_OTHER_DEFAULT_VARIABLE=foo

If necessary, we could also override variables defined in the .make/variables.env file, because the .make/.env is included last in the Makefile:

# File: Makefile
# ...

# include the default variables
include .make/variables.env
# include the local variables
-include .make/.env

The default value for ENVS is ENV=local TAG=latest to retain the same default behavior as before when ENVS is omitted. The corresponding make-init target is defined in the main Makefile and now looks like this:

ENVS?=ENV=local TAG=latest
.PHONY: make-init
make-init: ## Initializes the local .makefile/.env file with ENV variables for make. Use via ENVS="KEY_1=value1 KEY_2=value2"
    @$(if $(ENVS),,$(error ENVS is undefined))
    @rm -f .make/.env
    for variable in $(ENVS); do \
      echo $$variable | tee -a .make/.env > /dev/null 2>&1; \
    done
    @echo "Created a local .make/.env file"

ENV based `docker compose` config

As mentioned in section Compose file updates we need to select the "correct" docker compose configuration files based on the ENV value. This is done in .make/02-00-docker.mk:

# File .make/02-00-docker.mk

# ...

DOCKER_COMPOSE_DIR:=...
DOCKER_COMPOSE_COMMAND:=...

DOCKER_COMPOSE_FILE_LOCAL_CI:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.ci.yml
DOCKER_COMPOSE_FILE_CI:=$(DOCKER_COMPOSE_DIR)/docker-compose.ci.yml
DOCKER_COMPOSE_FILE_LOCAL:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.yml

# we need to "assemble" the correct combination of docker-compose.yml config files
ifeq ($(ENV),ci)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI) -f $(DOCKER_COMPOSE_FILE_CI)
else ifeq ($(ENV),local)
    DOCKER_COMPOSE_FILES:=-f $(DOCKER_COMPOSE_FILE_LOCAL_CI) -f $(DOCKER_COMPOSE_FILE_LOCAL)
endif

DOCKER_COMPOSE:=$(DOCKER_COMPOSE_COMMAND) $(DOCKER_COMPOSE_FILES)

When we now take a look at a full recipe when using ENV=ci with a docker target (e.g. docker-up), we can see that the correct files are chosen, e.g.

$ make docker-up ENV=ci -n
ENV=ci TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_ci --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.local.ci.yml -f ./.docker/docker-compose/docker-compose.ci.yml up -d

# =>
# -f ./.docker/docker-compose/docker-compose.local.ci.yml 
# -f ./.docker/docker-compose/docker-compose.ci.yml

Codebase changes

Add a test for encrypted files

We've introduced git-secret in the previous tutorial Use git-secret to encrypt secrets in the repository and used it to store the file passwords.txt encrypted in the codebase. To make sure that the decryption works as expected on the CI systems, I've added a test at tests/Feature/EncryptionTest.php to check if the file exists and if the content is correct.

class EncryptionTest extends TestCase
{
    public function test_ensure_that_the_secret_passwords_file_was_decrypted()
    {
        $pathToSecretFile = __DIR__."/../../passwords.txt";

        $this->assertFileExists($pathToSecretFile);

        $expected = "my_secret_password\n";
        $actual   = file_get_contents($pathToSecretFile);

        $this->assertEquals($expected, $actual);
    }
}

Of course this doesn't make sense in a "real world scenario", because the secret value would now be exposed in a test - but it suffices for now as proof of a working secret decryption.

Add a password-protected secret `gpg` key

I've mentioned in Scenario: Decrypt file that it is also possible to use a password-protected secret gpg key for an additional layer of security. I have created such a key and stored it in the repository at secret-protected.gpg.example (in a "real world scenario" I wouldn't do that - but since this is a public tutorial I want you to be able to follow along completely). The password for that key is 12345678.

The corresponding public key is located at .dev/gpg-keys/alice-protected-public.gpg and belongs to the email address alice.protected@example.com. I've added this email address and re-encrypted the secrets afterwards via

make gpg-init
make secret-add-user EMAIL="alice.protected@example.com"
make secret-encrypt

When I now import the secret-protected.gpg.example key, I can decrypt the secrets, though I cannot use the usual secret-decrypt target but must instead use secret-decrypt-with-password

make secret-decrypt-with-password GPG_PASSWORD=12345678

or store the GPG_PASSWORD in the .make/.env file when it is initialized for CI

make make-init ENVS="ENV=ci TAG=latest EXECUTE_IN_CONTAINER=true GPG_PASSWORD=12345678"
make secret-decrypt-with-password

Create a JUnit report from PhpUnit

I've added the --log-junit option to the phpunit configuration of the test make target in order to create an XML report in the .build/ directory in the .make/01-02-application-qa.mk file:

# File: .make/01-02-application-qa.mk
# ...

PHPUNIT_CMD=php vendor/bin/phpunit
PHPUNIT_ARGS= -c phpunit.xml --log-junit .build/report.xml

I.e. each run of the tests will now create a Junit XML report at .build/report.xml. The file is used as an example of a build artifact, i.e. "something that we would like to keep" from a CI run.

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now have a working CI pipeline for Github (via Github Actions) and/or Gitlab (via Gitlab pipelines) that runs automatically on each push.

In the next part of this tutorial, we will create a VM on GCP and provision it to run dockerized applications.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

Setting up Git Bash / MINGW / MSYS2 on Windows

Pascal Landau — Sat, 15 Apr 2023 12:25:38 +0000

An installation instruction for Git Bash / MINGW / MSYS2 on Windows with some notes on solving common problems

This article appeared first on https://www.pascallandau.com/ at Setting up Git Bash / MINGW / MSYS2 on Windows

In this article I'll document my process for setting up Git Bash / MINGW / MSYS2 on Windows including some additional configuration (e.g. installing make and apply some customizations via .bashrc).

Introduction
How to install and update Git Bash / MINGW / MSYS2 via Git for Windows
- Update MINGW
- How to install make
- Configuration via .bashrc
Common issues
- The role of winpty: Fixing "The input device is not a TTY"
- The path conversion issue
  - Fixing the path conversion issue for MINGW / MSYS2
  - Fixing the path conversion issue for winpty
Miscellaneous
- Change the bash custom prompt to a $

Introduction

When I was learning git I started with the fantastic Git for Windows package, that is maintained in the git-for-windows/git Github repository and comes with Git Bash, a shell that offers a Unix-terminal like experience. It uses MINGW and MSYS2 under the hood and does not only provide git but also a bunch of other common Linux utilities like

bash
sed
awk
ls
cp
rm
...

I believe the main "shell" is actually powered by MINGW64 as that's what will be shown by default:

Thus, I will refer to the tool as MINGW shell or Git Bash throughout this article.

I have been using MINGW for almost 10 years now, and it is still my go-to shell for Windows. I could just never warm up to WSL, because the file sharing performance between WSL and native Windows files was (is?) horrible - but that's a different story.

How to install and update Git Bash / MINGW / MSYS2 via Git for Windows

You can find the latest Git for Windows installation package directly at the homepage of https://gitforwindows.org/. Older releases can be found on Github in the Releases section of the git-for-windows/git repository

Follow the instructions in the How to Install Git Bash on Windows article on git-tower.com to get a guided tour through the setup process.

After the installation is finished, I usually create a desktop icon and assign the shortcut CTRL + ALT + B (for "bash") so that I can open a new shell session conveniently via keyboard.

Update MINGW

To update Git for Windows, you can simply run

git update-git-for-windows

Git for Windows comes with a tool to check for updates and offer to install them. Whether or
not you enabled auto-updates during installation, you can manually run
git update-git-for-windows.

You can check the current version via git version

$ git --version
git version 2.37.2.windows.2

Your browser does not support the video tag.

How to install `make`

As per How to add more to Git Bash on Windows: make:

Go to ezwinports
Download file make-4.3-without-guile-w32-bin.zip (get the version without guile)
Extract zip
Copy the contents to your Git/mingw64/ directory, merging the folders, but do NOT
overwrite/replace any existing files
- navigate to the Git/mingw64/ directory via
```
$(cd /; explorer .)
```

Test via make version

$ make --version
GNU Make 4.3.1
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Your browser does not support the video tag.

PS: There's also an alternative way that I've outlined in Install make on Windows (MinGW), though the one explained here is easier/faster.

Configuration via `.bashrc`

The MINGW shell is a bash shell and can thus be configured via a .bashrc file located at the home directory of the user. The shell supports the ~ character as an alias for the home directory, i.e. you can use ~/.bashrc to refer to the full path of the file. This means you can also edit it easily via vi ~/.bashrc - though I prefer an actual GUI editor like Notepad++. A common workflow for me to open the file is running the following commands in a MINGW shell session

# navigate to to the home directory
cd ~

# open the file explorer
explorer .

My .bashrc file usually includes the following setup:

# Get bash completion for make targets by parsing make files in the current directory at 
#  the file "Makefile"
#  all files with a ".mk" suffix in the folders ".make" and ".makefile"
# see https://stackoverflow.com/questions/4188324/bash-completion-of-makefile-target
# Notes:
#  -h hides filenames
#  -s hides error messages
complete -W "\`grep -shoE '^[a-zA-Z0-9_.-]+:([^=]|$)' Makefile .make/*.mk .makefile/*.mk | sed 's/[^a-zA-Z0-9_.-]*$//' | grep -v PHONY\`" make

# Docker login helper
# see https://www.pascallandau.com/blog/structuring-the-docker-setup-for-php-projects/#easy-container-access-via-din-bashrc-helper
function din() {
  filter=$1

  user=""
  if [[ -n "$2" ]];
  then
    user="--user $2"
  fi

  shell="bash"
  if [[ -n "$3" ]];
  then
    shell=$3
  fi

  prefix=""
  if [[ "$(expr substr $(uname -s) 1 5)" == "MINGW" ]]; then
    prefix="winpty"
  fi
  ${prefix} docker exec -it ${user} $(docker ps --filter name=${filter} -q | head -1) ${shell}
}

Links:

Common issues

The Git for Windows Known Issues page lists common problems with Git Bash and I want to provide some more context (and solutions) to the things that I have encountered.

The role of `winpty`: Fixing "The input device is not a TTY"

I encountered the The input device is not a TTY error while using docker. To log into a running docker container or starting a container with a login session, the -i (Keep STDIN open even if not attached) and -t (Allocate a pseudo-tty) options must be given:

For interactive processes (like a shell), you must use -i -t together in order to allocate a
tty for the container process. -i -t is often written -it.

But attempting to do so via

docker run --rm -it busybox sh

yields the following error:

$ docker run --rm -it busybox sh
the input device is not a TTY.  If you are using mintty, try prefixing the command with 'winpty'

Fortunately, the fix is included in the message: Prefix the command with winpty. Doing so works as expected:

$ winpty docker run --rm -it busybox sh
/ #

winpty is according to it's readme

[...] a Windows software package providing an interface similar to a Unix pty-master for
communicating with Windows console programs. The package consists of a library (libwinpty) and
a tool for Cygwin and MSYS for running Windows console programs in a Cygwin/MSYS pty.

So kind of a translator between your "Windows input" and the "command input" to create input that is compatible with a Unix pty (pty=pseudoterminal interface), e.g. for docker.

According to the Git for Windows Known Issues page, there are a number of other cases where winpty is required (though I personally didn't encounter them yet):

Some console programs, most notably non-MSYS2 Python, PHP, Node and OpenSSL, interact
correctly with MinTTY only when called through winpty (e.g. the Python console needs to be
started as winpty python instead of just python).

CAUTION: I've seen people put an alias in their .bashrc file to always prefix docker commands automatically with winpty like so:

alias docker="winpty docker"

However, winpty seems to break piping and can lead to unexpected results like the error stdout is not a tty. See the following example:

$ docker run --rm busybox echo "foo" | cat
foo

$ winpty docker run --rm busybox echo "foo" | cat
stdout is not a tty

You might work around this by adding the (undocumented) -Xallow-non-tty flag like so

$ winpty -Xallow-non-tty docker run --rm busybox echo "foo" | cat
foo

But this doesn't seem to be a catch-all solution and I would recommend against using it as a default - or if you do, only use it when the -it flag is used as proposed in this answer.

The path conversion issue

Ah. This one has given me lots of headaches over the years. MINGW, MSYS2 and winpty use automatic conversion of Unix paths to Windows paths, e.g. /foo gets translated to something like C:/Program Files/Git/foo where C:/Program Files/Git/ is the installation directory of the Git for Windows installation.

Fixing the path conversion issue for MINGW / MSYS2

First, the behavior is mentioned on the Git for Windows Known Issues page

If you specify command-line options starting with a slash, POSIX-to-Windows path conversion
will kick in converting e.g. "/usr/bin/bash.exe" to "C:\Program Files\Git\usr\bin\bash.exe".
When that is not desired -- e.g. "--upload-pack=/opt/git/bin/git-upload-pack" or "-L/regex/"
-- you need to set the environment variable MSYS_NO_PATHCONV temporarily, like so:

MSYS_NO_PATHCONV=1 git blame -L/pathconv/ msys2_path_conv.cc

Alternatively, you can double the first slash to avoid POSIX-to-Windows path conversion, e.g.
"//usr/bin/bash.exe".

and also documented for MINGW at "Posix path conversion", but it's still brought up regularly, see e.g. GH #3619: "/" is replaced with the directory path of Git installation when using MinGW64 Bash. or SO: How to stop MinGW and MSYS from mangling path names given at the command line

Example

$ docker run --rm busybox ls /foo
ls: C:/Program Files/Git/foo: No such file or directory

As quoted above, it can be solved by either

adding an additional / to the path

  $ docker run --rm busybox ls //foo
  ls: /foo: No such file or directory

prefixing the command with MSYS_NO_PATHCONV=1

  $ MSYS_NO_PATHCONV=1 docker run --rm busybox ls /foo
  ls: /foo: No such file or directory

or exporting the MSYS_NO_PATHCONV=1 variable as an environment variable to disable the behavior completely

  $ export MSYS_NO_PATHCONV=1
  $ docker run --rm busybox ls /foo
  ls: /foo: No such file or directory

CAUTION: The value of the MSYS_NO_PATHCONV variable does not matter - we can also set it to 0, false or an empty string. It only matters that the variable is defined!

$ MSYS_NO_PATHCONV=0 docker run --rm busybox ls /foo
ls: /foo: No such file or directory

This is particularly important when using the environment variable approach. In order to selectively enable the path conversion again, you must unset the MSYS_NO_PATHCONV first via env -u MSYS_NO_PATHCONV ..., e.g.

$ env -u MSYS_NO_PATHCONV docker run --rm busybox ls /foo
ls: C:/Program Files/Git/foo: No such file or directory

Your browser does not support the video tag.

CAUTION: I've seen people adding MSYS_NO_PATHCONV=1 permanently to their environment in their .bashrc file to always disable path conversion via

export MSYS_NO_PATHCONV=1

However, this can have some unintended side effects. When I tried it out, my local installation of the gcloud cli stopped working with the error

$ MSYS_NO_PATHCONV=1 gcloud version
C:\Users\Pascal\AppData\Local\Programs\Python\Python39\python.exe: can't open file 'C:\c\Users\Pascal\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\gcloud.py': [Errno 2] No such file or directory

So instead I recommend setting MSYS_NO_PATHCONV=1 either selectively per command or scope it to the use case. I do this for example in my Makefiles by only exporting it for the scope of make (and all scripts make invokes) by putting the following code in the beginning of the Makefile:

# OS is a defined variable for WIN systems, so "uname" will not be executed
OS?=$(shell uname)
# Values of OS:
#   Windows => Windows_NT 
#   Mac     => Darwin 
#   Linux   => Linux 
ifeq ($(OS),Windows_NT)
    export MSYS_NO_PATHCONV=1
endif

The path conversion is also documented for MSYS2 at "Filesystem Paths: Automatic Unix ⟶ Windows Path Conversion" and can be disabled via the MSYS2_ARG_CONV_EXCL environment variable:

[...] For these cases you can exclude certain arguments via the MSYS2_ARG_CONV_EXCL environment
variable:
[...]
MSYS2_ARG_CONV_EXCL can either be * to mean exclude everything, or a list of one ore more
arguments prefixes separated by ;, like MSYS2_ARG_CONV_EXCL=--dir=;--bla=;/test. It matches
the prefix against the whole argument string.

I.e. setting the variable as MSYS2_ARG_CONV_EXCL="*" should disable the path conversion completely. I myself have never had to use this, though. Using MSYS_NO_PATHCONV was always sufficient.

Fixing the path conversion issue for `winpty`

Unfortunately, winpty suffers from this path conversion issue as well. In the standard installation of Git for Windows we can even see this by simply using echo:

$ winpty echo /
C:/Program Files/Git

The behavior is known and flagged as a bug e.g. in GH issue #411: Path conversion with and without winpty differs.

Remember the example I gave in section The role of winpty e.g. when using docker?

$ winpty docker run --rm -it busybox sh
/ #

Now let's extend this and throw a volume into the mix:

$ winpty docker run --rm -v foo:/foo -it busybox sh
docker: Error response from daemon: create foo;C: "foo;C" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.

winpty converts /foo to C:/Program Files/Git/foo so that the volume definition becomes -v foo:C:/Program Files/Git/foo - which is of course invalid.

Using an additional / as a prefix does work here as well:

$ winpty docker run --rm -v foo://foo -it busybox sh
/ #

But there is no environment variable that we could use. The only way to "fix" the path conversion is using a newer release of winpty and replace the one that is shipped together with Git for Windows as proposed by the maintainer of winpty.

This comment outlines the full process to replace winpty and is (slightly adapted) as follows:

# create temporary directory
mkdir temp
cd temp
# download a newer release
curl -L https://github.com/rprichard/winpty/releases/download/0.4.3/winpty-0.4.3-msys2-2.7.0-x64.tar.gz --output winpty.tar.gz
# extract the archive
tar -xvf winpty.tar.gz
# copy the content of the bin/ folder to `/usr/bin` 
# (which resolves to e.g `C:/Program Files/Git/usr/bin`; replaces any existing files)
cp winpty-0.4.3-msys2-2.7.0-x64/bin/* /usr/bin
# delete the temporary directory
cd ..
rm -rf temp/

Your browser does not support the video tag.

Once the new version is installed, the path conversion does not happen any longer (even without specifying any environment variables).

Related comments:

Caution
After updating MinGW, the fix for winpty is "gone"!

Your browser does not support the video tag.

I.e. you need to re-run the steps above every time you run an update.

Miscellaneous

Some stuff that I need from time to time - not necessarily only relevant for Git Bash.

Change the `bash` custom prompt to a `$`

Via PS1=" $":

Pascal@LAPTOP-0DNL2Q02 MINGW64 ~
$ PS1="$ "
$

See How to Change / Set up bash custom prompt (PS1) in Linux

Use git-secret to encrypt secrets in the repository [Tutorial Part 6]

Pascal Landau — Mon, 04 Jul 2022 05:00:51 +0000

How to use git-secret to encrypt secrets and store them in a git repository

This article appeared first on https://www.pascallandau.com/ at Use git-secret to encrypt secrets in the repository [Tutorial Part 6]

In the sixth part of this tutorial series on developing PHP on Docker we will setup git-secret to store secrets directly in the repository. Everything will be handled through Docker and added as make targets for a convenient workflow.

FYI: This tutorial is a precursor to the next a part
Create a CI pipeline for dockerized PHP Apps
because dealing with secrets is an important aspect when setting up a CI system (and later when deploying to production) - but I feel it's complex enough to warrant its own article.

All code samples are publicly available in my Docker PHP Tutorial repository on Github. You find the branch with the final result of this tutorial at part-6-git-secret-encrypt-repository-docker.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Set up PHP QA tools and control them via make and the following one is Create a CI pipeline for dockerized PHP Apps.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
Tooling
- gpg
  - gpg installation
  - gpg usage
    - Create GPG key pair
    - Export, list and import private GPG keys
    - Export, list and import public GPG keys
- git-secret
  - git-secret installation
    - The git permission issue
  - git-secret usage
    - Initialize git-secret
    - Adding, listing and removing users
    - Adding, listing and removing files for encryption
    - Encrypt files
    - Decrypting files
    - Show changes between encrypted and decrypted files
Makefile adjustments
Workflow
- Process challenges
  - Updating secrets
  - Code reviews and merge conflicts
  - Local git-secret and gpg setup
- Scenarios
  - Initial setup of gpg keys
  - Initial setup of git-secret
  - Initialize gpg after container startup
  - Adding (new) team members
  - Adding and encrypting files
  - Decrypt files
  - Removing files
  - Removing team members
Pros and cons
- Pro
- Cons
Wrapping up

Introduction

Dealing with secrets (passwords, tokens, key files, etc.) is close to "naming things" when it comes to hard problems in software engineering. Some things to consider:

security is paramount - but high security often goes hand in hand with high inconvenience
- and if things get too complicated, people look for shortcuts...
in a team, sharing certain secret values is often mandatory
- so now we need to think about secure ways to distribute and update secrets across multiple people
concrete secret values often depend on the environment
- inherently tricky to "test" or even "review", because those values are "by definition" different on "your machine" than on "production"

In fact, entire products have been build around dealing with secrets, e.g. HashiCorp Vault, AWS Secrets Manager or the GCP Secret Manager. Introducing those in a project comes with a certain overhead as it's yet another service that needs to be integrated and maintained. Maybe it is the exactly right decision for your use-case - maybe it's overkill. By the end of this article you'll at least be aware of an alternative with a lower barrier to entry. See also the Pros and cons section in the end for an overview.

Even though it's generally not advised to store secrets in a repository, I'll propose exactly that in this tutorial:

identify files that contain secret values
make sure they are added to .gitignore
encrypt them via git-secret
commit the encrypted files to the repository

In the end, we will be able to call

make secret-decrypt

to reveal secrets in the codebase, make modifications to them if necessary and then run

make secret-encrypt

to encrypt them again so that they can be committed (and pushed to the remote repository). To see it in action, check out branch part-6-git-secret-encrypt-repository-docker and run the following commands:

# checkout the branch
git checkout part-6-git-secret-encrypt-repository-docker

# build and start the docker setup
make make-init
make docker-build
make docker-up

# "create" the secret key - the file "secret.gpg.example" would usually NOT live in the repo!
cp secret.gpg.example secret.gpg

# initialize gpg
make gpg-init

# ensure that the decrypted secret file does not exist
ls passwords.txt

# decrypt the secret file
make secret-decrypt

# show the content of the secret file
cat passwords.txt

Tooling

We will set up gpg and git-secret in the php base image, so that the tools become available in all other containers. Please refer to Docker from scratch for PHP 8.1 Applications in 2022 for an in-depth explanation of the docker images.

<strong>Caution</strong>


All following commands are 
<strong>executed <em>in</em> the <code>application</code> container.</strong>
<br>
<br>
<strong>Tip:</strong>
<br>
See <a href="https://www.pascallandau.com/blog/structuring-the-docker-setup-for-php-projects/#easy-container-access-via%20-din-bashrc-helper">Easy container access via din .bashrc helper</a>
for a convenient shortcut to log into docker containers.

Please note, that there is a caveat when using git-secret in a folder that is shared between the host system and a docker container. I'll explain that in more detail (including a workaround) in section The git-secret directory and the gpg-agent socket.

gpg

gpg is short for The GNU Privacy Guard and is an open source implementation of the OpenPGP standard. In short, it allows us to create a personal key file pair (similar to SSH keys) with a private secret key and a public key that can be shared with other parties whose messages you want to decrypt.

gpg installation

To install it, we can simply run apk add gnupg and thus update .docker/images/php/base/Dockerfile accordingly

# File: .docker/images/php/base/Dockerfile

RUN apk add --update --no-cache \
        bash \
        gnupg \
        make \
#...

gpg usage

I'll only cover the strictly necessary gpg commands here. Please refer to the "Using GPG" section in the git-secret docu and/or How to generate PGP keys with GPG for further information.

Create GPG key pair

We need gpg to create the gpg key pair via

name="Pascal Landau"
email="pascal.landau@example.com"
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
%no-protection
EOF

The %no-protection will create a key without password, see also this gist to "Creating gpg keys non-interactively". To use a password (e.g. 12345678, we could have replace the %no-protection line with

Passphrase: 12345678

All options for the unattended creation are defined in the official docs at "Unattended key generation".

Output:

$ name="Pascal Landau"
$ email="pascal.landau@example.com"
$ gpg --batch --gen-key <<EOF
> Key-Type: 1
> Key-Length: 2048
> Subkey-Type: 1
> Subkey-Length: 2048
> Name-Real: $name
> Name-Email: $email
> Expire-Date: 0
> %no-protection
> EOF
gpg: key E1E734E00B611C26 marked as ultimately trusted
gpg: revocation certificate stored as '/root/.gnupg/opengpg-revocs.d/74082D81525723F5BF5B2099E1E734E00B611C26.rev'

You could also run gpg --gen-key without the --batch flag to be guided interactively through the process.

Export, list and import private GPG keys

The private key can be exported via

email="pascal.landau@example.com"
path="secret.gpg"
gpg --output "$path" --armor --export-secret-key "$email"

This secret key must never be shared!

It looks like this:

All secret keys can be listed via

gpg --list-secret-keys

Output:

$ gpg --list-secret-keys
/root/.gnupg/pubring.kbx
------------------------
sec   rsa2048 2022-03-27 [SCEA]
      74082D81525723F5BF5B2099E1E734E00B611C26
uid           [ultimate] Pascal Landau <pascal.landau@example.com>
ssb   rsa2048 2022-03-27 [SEA]

You can import the private key via

path="secret.gpg"
gpg --import "$path"

and get the following output:

$ path="secret.gpg"
$ gpg --import "$path"
gpg: key E1E734E00B611C26: "Pascal Landau <pascal.landau@example.com>" not changed
gpg: key E1E734E00B611C26: secret key imported
gpg: Total number processed: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:  secret keys unchanged: 1

Caution: If the secret key requires a password, you would now be prompted for it. We can circumvent the prompt by using --batch --yes --pinentry-mode loopback:

path="secret.gpg"
gpg --import --batch --yes --pinentry-mode loopback "$path"

See also Using Command-Line Passphrase Input for GPG. In doing so, we don't need to provide the password just yet - but we must pass it later when we attempt to decrypt files.

Export, list and import public GPG keys

The public key can be exported to public.gpg via

email="pascal.landau@example.com"
path="public.gpg"
gpg --armor --export "$email" > "$path"

It looks like this:

List all public keys via

gpg --list-keys

Output:

$ gpg --list-keys
/root/.gnupg/pubring.kbx
------------------------
pub   rsa2048 2022-03-27 [SCEA]
      74082D81525723F5BF5B2099E1E734E00B611C26
uid           [ultimate] Pascal Landau <pascal.landau@example.com>
sub   rsa2048 2022-03-27 [SEA]

The public key can be imported in the same way as private keys via

path="public.gpg"
gpg --import "$path"

Example:

$ gpg --import /var/www/app/public.gpg
gpg: key E1E734E00B611C26: "Pascal Landau <pascal.landau@example.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

git-secret

The official website of git-secret is already doing a great job of introducing the tool. In short, it allows us to declare certain files as "secrets" and encrypt them via gpg - using the keys of all trusted parties. The encrypted file can then by stored safely directly in the git repository and decrypted if required.

In this tutorial I'm using git-secret v0.4.0

$ git secret --version
0.4.0

git-secret installation

The installation instructions for Alpine read as follows:

sh -c "echo 'https://gitsecret.jfrog.io/artifactory/git-secret-apk/all/main'" >> /etc/apk/repositories
wget -O /etc/apk/keys/git-secret-apk.rsa.pub 'https://gitsecret.jfrog.io/artifactory/api/security/keypair/public/repositories/git-secret-apk'
apk add --update --no-cache git-secret

Plus, we need to account for a recent change in git that requires that the parent directory is owned by the user executing the git command. See also the more detailed explanation in section The git permission issue.

We update the .docker/images/php/base/Dockerfile accordingly:

# File: .docker/images/php/base/Dockerfile

# install git-secret
# @see https://git-secret.io/installation#alpine
ADD https://gitsecret.jfrog.io/artifactory/api/security/keypair/public/repositories/git-secret-apk /etc/apk/keys/git-secret-apk.rsa.pub

RUN echo "https://gitsecret.jfrog.io/artifactory/git-secret-apk/all/main" >> /etc/apk/repositories  && \
    apk add --update --no-cache \
        bash \
        git-secret \
        gawk \
        gnupg \
        make \
#...

# Fix the git permission issue
RUN git config --system --add safe.directory "$APP_CODE_PATH"

The `git` permission issue

In April 2022, Github accounced the security vulnerability CVE-2022-24765, that was fixed in git v2.35.2

This version changes Git’s behavior when looking for a top-level .git directory to stop when
its directory traversal changes ownership from the current user.

In practice, the following error occurs if the parent directory is not owned by the user that executes the git command

Error: fatal: unsafe repository ('/parent/dir/of/.git-folder' is owned by someone else)
To add an exception for this directory, call:

    git config --global --add safe.directory /parent/dir/of/.git-folder

When using git secret, we would get the slightly misleading error message

git-secret: abort: not in dir with git repo. Use 'git init' or 'git clone', then in repo use 'git secret init'

We can "fix" the issue by using the new multi-valued safe.directory configuration via

git config --system --add safe.directory /parent/dir/of/.git-folder

Note, that we didn't use the suggested --global option but --system instead, so that the configuration is set for any user.

Wait - why not just ensure that the parent directory of the .git folder has the correct permissions?

Well... there's currently (2022-05-28) a bug in Docker Desktop that makes the permissions of bind mounts kinda unpredictable, see Ownership of files set via bind mount is set to user who accesses the file first and by applying the fix directly in the Dockerfile we can solve the issue reliably.

git-secret usage

Initialize git-secret

git-secret is initialized via the following command run in the root of the git repository

git secret init

$ git secret init
git-secret: init created: '/var/www/app/.gitsecret/'

We only need to do this once, because we'll commit the folder to git later. It contains the following files:

$ git status | grep ".gitsecret"
        new file:   .gitsecret/keys/pubring.kbx
        new file:   .gitsecret/keys/pubring.kbx~
        new file:   .gitsecret/keys/trustdb.gpg
        new file:   .gitsecret/paths/mapping.cfg

The pubring.kbx~ file (with the trailing tilde ~) is only a temporary file and can safely be git-ignored. See also Can't find any docs about keyring.kbx~ file.

The `git-secret` directory and the `gpg-agent` socket

To use git-secret in a directory that is shared between the host system and docker, we need to also run the following commands:

tee .gitsecret/keys/S.gpg-agent <<EOF
%Assuan%
socket=/tmp/S.gpg-agent
EOF

tee .gitsecret/keys/S.gpg-agent.ssh <<EOF
%Assuan%
socket=/tmp/S.gpg-agent.ssh
EOF

tee .gitsecret/keys/gpg-agent.conf <<EOF
extra-socket /tmp/S.gpg-agent.extra
browser-socket /tmp/S.gpg-agent.browser
EOF

This is necessary because there is an issue when git-secret is used in a setup where the codebase is shared between the host system and a docker container. I've explained the details in the Github issue "gpg: can't connect to the agent: IPC connect call failed" error in docker alpine on shared volume.

In short:

gpg uses a gpg-agent to perform its tasks and the two tools communicate through sockets that are created in the --home-directory of the gpg-agent
the agent is started implicitly through a gpg command used by git-secret, using the .gitsecret/keys directories as a --home-directory
because the location of the --home-directory is shared with the host system, the socket creation fails (potentially only an issue for Docker Desktop, see the related discussion in Github issue Support for sharing unix sockets) The corresponding error messages are

gpg: can't connect to the agent: IPC connect call failed

gpg-agent: error binding socket to '/var/www/app/.gitsecret/keys/S.gpg-agent': I/O error

The workaround for this problem can be found in this thread: Configure gpg to use different locations for the sockets by placing additional gpg configuration files in the .gitsecret/keys directory:

S.gpg-agent

%Assuan%
socket=/tmp/S.gpg-agent

S.gpg-agent.ssh

%Assuan%
socket=/tmp/S.gpg-agent.ssh

gpg-agent.conf

extra-socket /tmp/S.gpg-agent.extra
browser-socket /tmp/S.gpg-agent.browser

Adding, listing and removing users

To add a new user, you must first import its public gpg key. Then run:

email="pascal.landau@example.com"
git secret tell "$email"

In this case, the user pascal.landau@example.com will now be able to decrypt the secrets.

To show the users run

git secret whoknows

$ git secret whoknows
pascal.landau@example.com

To remove a user, run

email="pascal.landau@example.com"
git secret killperson "$email"

FYI: This command was renamed to removeperson in git-secret >= 0.5.0

$ git secret killperson pascal.landau@example.com
git-secret: removed keys.
git-secret: now [pascal.landau@example.com] do not have an access to the repository.
git-secret: make sure to hide the existing secrets again.

User pascal.landau@example.com will no longer be able to decrypt the secrets.

Caution: The secrets need to be re-encrypted after removing a user!

Reminder: Rotate the encrypted secrets

Please be aware that not only your secrets are stored in git, but who had access as well. I.e. even if you remove a user and re-encrypt the secrets, that user would still be able to decrypt the secrets of a previous commit (when the user was still added). In consequence, you need to rotate the encrypted secrets themselves as well after removing a user.

But isn't that a great flaw in the system, making it a bad idea to use git-secret in general?

In my opinion: No.

If the removed user had access to the secrets at any point in time (no matter where they have been stored), he could very well have just created a local copy or simply "written them down". In terms of security there is really no "added downside" due to git-secret. It just makes it very clear that you must rotate the secrets ¯\_(ツ)_/¯

Adding, listing and removing files for encryption

Run git secret add [filenames...] for files you want to encrypt. Example:

git secret add .env

If .env is not added in .gitignore, git-secret will display a warning and add it automatically.

git-secret: these files are not in .gitignore: .env
git-secret: auto adding them to .env
git-secret: 1 item(s) added.

Otherwise, the file is added with no warning.

$ git secret add .env
git-secret: 1 item(s) added.

You only need to add files once. They are then stored in .gitsecret/paths/mapping.cfg:

$ cat .gitsecret/paths/mapping.cfg
.env:505070fc20233cb426eac6a3414399d0f466710c993198b1088e897fdfbbb2d5

You can also show the added files via

git secret list

$ git secret list
.env

Caution: The files are not yet encrypted!

If you want to remove a file from being encrypted, run

git secret remove .env

Output

$ git secret remove .env
git-secret: removed from index.
git-secret: ensure that files: [.env] are now not ignored.

Encrypt files

To actually encrypt the files, run:

git secret hide

Output:

$ git secret hide
git-secret: done. 1 of 1 files are hidden.

The encrypted (binary) file is stored at $filename.secret, i.e. .env.secret in this case:

$ cat .env.secret
�☺♀♥�H~�B�Ӯ☺�"��▼♂F�►���l�Cs��S�@MHWs��e������{♣♫↕↓�L� ↕s�1�J$◄♥�;���ǆ֕�Za�����\u�ٲ& ¶��V�► ���6��
;<�d:��}ҨD%.�;��&��G����vWW�]>���߶��▲;D�+Rs�S→�Y!&J��۪8���ٔF��→f����*��$♠���&RC�8▼♂�☻z h��Z0M�T>

The encrypted files are de-cryptable for all users that have been added via git secret tell. That also means that you need to run this command again whenever a new user is added.

Decrypting files

You can decrypt files via

git secret reveal

Output:

$ git secret reveal
File '/var/www/app/.env' exists. Overwrite? (y/N) y
git-secret: done. 1 of 1 files are revealed.

the files are decrypted and will overwrite the current, unencrypted files (if they already exist)
- use the -f option to force the overwrite and run non-interactively
if you only want to check the content of an encrypted file, you can use git secret cat $filename (e.g. git secret cat .env)

In case the secret gpg key is password protected, you must pass the password via the -p option. E.g. for password 123456

git secret reveal -p 123456

Show changes between encrypted and decrypted files

One problem that comes with encrypted files: You can't review them during a code review in a remote tool. So in order to understand what changes have been made, it is helpful to
show the changes between the encrypted and the decrypted files. This can be done via

git secret changes

Output:

$ echo "foo" >> .env
$ git secret changes
git-secret: changes in /var/www/app/.env:
--- /dev/fd/63
+++ /var/www/app/.env
@@ -34,3 +34,4 @@
 MAIL_ENCRYPTION=null
 MAIL_FROM_ADDRESS=null
 MAIL_FROM_NAME="${APP_NAME}"
+foo

Note the +foo at the bottom of the output. It was added in the first line via echo "foo"> >> .env.

Makefile adjustments

Since I won't be able to remember all the commands for git-secret and gpg, I've added them to the Makefile at .make/01-00-application-setup.mk:

# File: .make/01-00-application-setup.mk

#...

# gpg

DEFAULT_SECRET_GPG_KEY?=secret.gpg
DEFAULT_PUBLIC_GPG_KEYS?=.dev/gpg-keys/*

.PHONY: gpg
gpg: ## Run gpg commands. Specify the command e.g. via ARGS="--list-keys"
    $(EXECUTE_IN_APPLICATION_CONTAINER) gpg $(ARGS)

.PHONY: gpg-export-public-key
gpg-export-public-key: ## Export a gpg public key e.g. via EMAIL="john.doe@example.com" PATH=".dev/gpg-keys/john-public.gpg"
    @$(if $(PATH),,$(error PATH is undefined))
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s gpg ARGS="gpg --armor --export $(EMAIL) > $(PATH)"

.PHONY: gpg-export-private-key
gpg-export-private-key: ## Export a gpg private key e.g. via EMAIL="john.doe@example.com" PATH="secret.gpg"
    @$(if $(PATH),,$(error PATH is undefined))
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s gpg ARGS="--output $(PATH) --armor --export-secret-key $(EMAIL)"

.PHONY: gpg-import
gpg-import: ## Import a gpg key file e.g. via GPG_KEY_FILES="/path/to/file /path/to/file2"
    @$(if $(GPG_KEY_FILES),,$(error GPG_KEY_FILES is undefined))
    "$(MAKE)" -s gpg ARGS="--import --batch --yes --pinentry-mode loopback $(GPG_KEY_FILES)"

.PHONY: gpg-import-default-secret-key
gpg-import-default-secret-key: ## Import the default secret key
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_SECRET_GPG_KEY)"

.PHONY: gpg-import-default-public-keys
gpg-import-default-public-keys: ## Import the default public keys
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_PUBLIC_GPG_KEYS)" 

.PHONY: gpg-init
gpg-init: gpg-import-default-secret-key gpg-import-default-public-keys ## Initialize gpg in the container, i.e. import all public and private keys

# git-secret

.PHONY: git-secret
git-secret: ## Run git-secret commands. Specify the command e.g. via ARGS="hide"
    $(EXECUTE_IN_APPLICATION_CONTAINER) git-secret $(ARGS)

.PHONY: secret-init
secret-init: ## Initialize git-secret in the repository via `git-secret init`
    "$(MAKE)" -s git-secret ARGS="init"

.PHONY: secret-init-gpg-socket-config
secret-init-gpg-socket-config: ## Initialize the config files to change the gpg socket locations
    echo "%Assuan%" > .gitsecret/keys/S.gpg-agent
    echo "socket=/tmp/S.gpg-agent" >> .gitsecret/keys/S.gpg-agent
    echo "%Assuan%" > .gitsecret/keys/S.gpg-agent.ssh
    echo "socket=/tmp/S.gpg-agent.ssh" >> .gitsecret/keys/S.gpg-agent.ssh
    echo "extra-socket /tmp/S.gpg-agent.extra" > .gitsecret/keys/gpg-agent.conf
    echo "browser-socket /tmp/S.gpg-agent.browser" >> .gitsecret/keys/gpg-agent.conf

.PHONY: secret-encrypt
secret-encrypt: ## Decrypt secret files via `git-secret hide`
    "$(MAKE)" -s git-secret ARGS="hide"

.PHONY: secret-decrypt
secret-decrypt: ## Decrypt secret files via `git-secret reveal -f`
    "$(MAKE)" -s git-secret ARGS="reveal -f" 

.PHONY: secret-decrypt-with-password
secret-decrypt-with-password: ## Decrypt secret files using a password for gpg via `git-secret reveal -f -p $(GPG_PASSWORD)`
    @$(if $(GPG_PASSWORD),,$(error GPG_PASSWORD is undefined))
    "$(MAKE)" -s git-secret ARGS="reveal -f -p $(GPG_PASSWORD)" 

.PHONY: secret-add
secret-add: ## Add a file to git secret via `git-secret add $FILE`
    @$(if $(FILE),,$(error FILE is undefined))
    "$(MAKE)" -s git-secret ARGS="add $(FILE)"

.PHONY: secret-cat
secret-cat: ## Show the contents of file to git secret via `git-secret cat $FILE`
    @$(if $(FILE),,$(error FILE is undefined))
    "$(MAKE)" -s git-secret ARGS="cat $(FILE)"

.PHONY: secret-list
secret-list: ## List all files added to git secret `git-secret list`
    "$(MAKE)" -s git-secret ARGS="list"

.PHONY: secret-remove
secret-remove: ## Remove a file from git secret via `git-secret remove $FILE`
    @$(if $(FILE),,$(error FILE is undefined))
    "$(MAKE)" -s git-secret ARGS="remove $(FILE)"

.PHONY: secret-add-user
secret-add-user: ## Remove a user from git secret via `git-secret tell $EMAIL`
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s git-secret ARGS="tell $(EMAIL)"

.PHONY: secret-show-users
secret-show-users: ## Show all users that have access to git secret via `git-secret whoknows`
    "$(MAKE)" -s git-secret ARGS="whoknows"

.PHONY: secret-remove-user
secret-remove-user: ## Remove a user from git secret via `git-secret killperson $EMAIL`
    @$(if $(EMAIL),,$(error EMAIL is undefined))
    "$(MAKE)" -s git-secret ARGS="killperson $(EMAIL)"

.PHONY: secret-diff
secret-diff: ## Show the diff between the content of encrypted and decrypted files via `git-secret changes`
    "$(MAKE)" -s git-secret ARGS="changes"

Workflow

Working with git-secret is pretty straight forward:

initialize git-secret
add all users
add all secret files and make sure they are ignored via .gitignore
encrypt the files
commit the encrypted files like "any other file"
if any changes were made by other team members to the files:
- => decrypt to get the most up-to-date ones
if any modifications are required from your side:
- => make the changes to the decrypted files and then re-encrypt them again

But: The devil is in the details. The Process challenges section explains some of the pitfalls that we have encountered and the Scenarios section gives some concrete examples for common scenarios.

Process challenges

From a process perspective we've encountered some challenges that I'd like to mention - including how we deal with them.

Updating secrets

When updating secrets you must ensure to always decrypt the files first in order to avoid using "stale" files that you might still have locally. I usually check out the latest main branch and run git secret reveal to have the most up-to-date versions of the secret files. You could also use a post-merge git hook to do this automatically, but I personally don't want to risk overwriting my local secret files by accident.

Code reviews and merge conflicts

Since the encrypted files cannot be diffed meaningfully, the code reviews become more difficult when secrets are involved. We use Gitlab for reviews and I usually first check the diff of the .gitsecret/paths/mapping.cfg file to see "which files have changed" directly in the UI.

In addition, I will

checkout the main branch
decrypt the secrets via git secret reveal -f
checkout the feature-branch
run git secret changes to see the differences between the decrypted files from main and the encrypted files from feature-branch

Things get even more complicated when multiple team members need to modify secret files at the same time on different branches, as the encrypted files cannot be compared - i.e. git cannot be smart about delta updates. The only way around this is coordinating the pull requests, i.e. merge the first, update the secrets of the second and then merge the second.

Fortunately, this has only happened very rarely so far.

Local `git-secret` and `gpg` setup

Currently, all developers in our team have git-secret installed locally (instead of using it through docker) and use their own gpg keys.

This means more onboarding overhead, because

a new dev must
- install git-secret locally (*)
- install and setup gpg locally (*)
- create a gpg key pair
the public key must be added by every other team member (*)
the user of the key must be added via git secret tell
the secrets must be re-encrypted

And for offboarding

the public key must be removed by every other team member (*)
the user of the key must be removed via git secret killperson
the secrets must be re-encrypted

Plus, we need to ensure that the git-secret and gpg versions are kept up-to-date for everyone to not run into any compatibility issues.

As an alternative, I'm currently leaning more towards handling everything through docker (as presented in this tutorial). All steps marked with (*) are then obsolete, i.e. there is no need to setup git-secret and gpg locally.

But the approach also comes with some downsides, because

the secret key and all public keys have to be imported every time the container is started
each dev needs to put his private gpg key "in the codebase" (ignored by .gitignore) so it can be shared with docker and imported by gpg (in docker). The alternative would be using a single secret key that is shared within the team - which feels very wrong :P

To make this a little more convenient, we put the public gpg keys of every dev in the repository under .dev/gpg-keys/ and the private key has to be named secret.gpg and put in the root of the codebase.

In this setup, secret.gpg must also be added to the.gitignore file.

# File: .gitignore
#...
vendor/
secret.gpg

The import can now be simplified with make targets:

# gpg

DEFAULT_SECRET_GPG_KEY?=secret.gpg
DEFAULT_PUBLIC_GPG_KEYS?=.dev/gpg-keys/*

.PHONY: gpg
gpg: ## Run gpg commands. Specify the command e.g. via ARGS="--list-keys"
    $(EXECUTE_IN_APPLICATION_CONTAINER) gpg $(ARGS)

.PHONY: gpg-import
gpg-import: ## Import a gpg key file e.g. via GPG_KEY_FILES="/path/to/file /path/to/file2"
    @$(if $(GPG_KEY_FILES),,$(error GPG_KEY_FILES is undefined))
    "$(MAKE)" -s gpg ARGS="--import --batch --yes --pinentry-mode loopback $(GPG_KEY_FILES)"

.PHONY: gpg-import-default-secret-key
gpg-import-default-secret-key: ## Import the default secret key
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_SECRET_GPG_KEY)"

.PHONY: gpg-import-default-public-keys
gpg-import-default-public-keys: ## Import the default public keys
    "$(MAKE)" -s gpg-import GPG_KEY_FILES="$(DEFAULT_PUBLIC_GPG_KEYS)" 

.PHONY: gpg-init
gpg-init: gpg-import-default-secret-key gpg-import-default-public-keys ## Initialize gpg in the container, i.e. import all public and private keys

"Everything" can now be handled via

make gpg-init

that needs to be run one single time after a container has been started.

Scenarios

The scenarios assume the following preconditions:

You have checked out branch part-6-git-secret-encrypt-repository-docker

  git checkout part-6-git-secret-encrypt-repository-docker

and no running docker containers

  make docker-down

You have deleted the existing git-secret folder, the keys in .dev/gpg-keys, the secret.gpg key and the passwords.* files

  rm -rf .gitsecret/ .dev/gpg-keys/* secret.gpg passwords.*

Initial setup of `gpg` keys

Unfortunately, I didn't find a way to create and export gpg keys through make and docker. You need to either run the commands interactively OR pass a string with newlines to it. Both things are horribly complicated with make and docker. Thus, you need to log into the application container and run the commands in there directly. Not great - but this needs to be done only once when a new developer is onboarded anyways.

FYI: I usually log into containers via Easy container access via din .bashrc helper.

The secret key is exported to secret.gpg and the public key to .dev/gpg-keys/alice-public.gpg.

# start the docker setup
make docker-up

# log into the container ('winpty' is only required on Windows)
winpty docker exec -ti dofroscra_local-application-1 bash

# export key pair
name="Alice Doe"
email="alice@example.com"
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
%no-protection
EOF

# export the private key
gpg --output secret.gpg --armor --export-secret-key $email

# export the public key
gpg --armor --export $email > .dev/gpg-keys/alice-public.gpg

$ make docker-up
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml up -d
Container dofroscra_local-application-1  Created
...
Container dofroscra_local-application-1  Started
$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS          PORTS                NAMES
...
95f740607586   dofroscra/application-local:latest   "/usr/sbin/sshd -D"      21 minutes ago   Up 21 minutes   0.0.0.0:2222->22/tcp dofroscra_local-application-1

$ winpty docker exec -ti dofroscra_local-application-1 bash
root:/var/www/app# name="Alice Doe"
root:/var/www/app# email="alice@example.com"
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: $name
Name-Email: $email
Expire-Date: 0
%no-protection
EOF
root:/var/www/app# gpg --batch --gen-key <<EOF
> Key-Type: 1
> Key-Length: 2048
> Subkey-Type: 1
> Subkey-Length: 2048
> Name-Real: $name
> Name-Email: $email
> Expire-Date: 0
> %no-protection
> EOF
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key BBBE654440E720C1 marked as ultimately trusted
gpg: directory '/root/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/225C736E0E70AC222C072B70BBBE654440E720C1.rev'

root:/var/www/app# gpg --output secret.gpg --armor --export-secret-key $email
root:/var/www/app# head secret.gpg
-----BEGIN PGP PRIVATE KEY BLOCK-----

lQOYBGJD+bwBCADBGKySV5PINc5MmQB3PNvCG7Oa1VMBO8XJdivIOSw7ykv55PRP
3g3R+ERd1Ss5gd5KAxLc1tt6PHGSPTypUJjCng2plwD8Jy5A/cC6o2x8yubOslLa
x1EC9fpcxUYUNXZavtEr+ylOaTaRz6qwSabsAgkg2NZ0ey/QKmFOZvhL8NlK9lTI
GgZPTiqPCsr7hiNg0WRbT5h8nTmfpl/DdTgwfPsDn5Hn0TEMa79WsrPnnq16jsq0
Uusuw3tOmdSdYnT8j7m1cpgcSj0hRF1eh4GVE0o62GqeLTWW9mfpcuv7n6mWaCB8
DCH6H238gwUriq/aboegcuBktlvSY21q/MIXABEBAAEAB/wK/M2buX+vavRgDRgR
hjUrsJTXO3VGLYcIetYXRhLmHLxBriKtcBa8OxLKKL5AFEuNourOBdcmTPiEwuxH
5s39IQOTrK6B1UmUqXvFLasXghorv8o8KGRL4ABM4Bgn6o+KBAVLVIwvVIhQ4rlf

root:/var/www/app# gpg --armor --export $email > .dev/gpg-keys/alice-public.gpg
root:/var/www/app# head .dev/gpg-keys/alice-public.gpg
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQENBGJD+bwBCADBGKySV5PINc5MmQB3PNvCG7Oa1VMBO8XJdivIOSw7ykv55PRP
3g3R+ERd1Ss5gd5KAxLc1tt6PHGSPTypUJjCng2plwD8Jy5A/cC6o2x8yubOslLa
x1EC9fpcxUYUNXZavtEr+ylOaTaRz6qwSabsAgkg2NZ0ey/QKmFOZvhL8NlK9lTI
GgZPTiqPCsr7hiNg0WRbT5h8nTmfpl/DdTgwfPsDn5Hn0TEMa79WsrPnnq16jsq0
Uusuw3tOmdSdYnT8j7m1cpgcSj0hRF1eh4GVE0o62GqeLTWW9mfpcuv7n6mWaCB8
DCH6H238gwUriq/aboegcuBktlvSY21q/MIXABEBAAG0HUFsaWNlIERvZSA8YWxp
Y2VAZXhhbXBsZS5jb20+iQFOBBMBCgA4FiEEIlxzbg5wrCIsBytwu75lREDnIMEF
AmJD+bwCGy8FCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQu75lREDnIMEN4Af+

That's it. We now have a new secret and private key for alice@example.com and have exported it to secret.gpg resp. .dev/gpg-keys/alice-public.gpg (and thus shared it with the host system). The remaining commands can now be run outside of the application container directly on the host system.

Initial setup of `git-secret`

Let's say we want to introduce git-secret "from scratch" to a new codebase. Then you would run the following commands:

Initialize git-secret

make secret-init

$ make secret-init
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="init";
git-secret: init created: '/var/www/app/.gitsecret/'

Apply the gpg fix for shared directories

See The git-secret directory and the gpg-agent socket.

$ make secret-init-gpg-socket-config

$ make secret-init-gpg-socket-config
echo "%Assuan%" > .gitsecret/keys/S.gpg-agent
echo "socket=/tmp/S.gpg-agent" >> .gitsecret/keys/S.gpg-agent
echo "%Assuan%" > .gitsecret/keys/S.gpg-agent.ssh
echo "socket=/tmp/S.gpg-agent.ssh" >> .gitsecret/keys/S.gpg-agent.ssh
echo "extra-socket /tmp/S.gpg-agent.extra" > .gitsecret/keys/gpg-agent.conf
echo "browser-socket /tmp/S.gpg-agent.browser" >> .gitsecret/keys/gpg-agent.conf

Initialize `gpg` after container startup

After restarting the containers, we need to initialize gpg, i.e. import all public keys from .dev/gpg-keys/* and the private key from secret.gpg. Otherwise we will not be able to en- and decrypt the files.

make gpg-init

$ make gpg-init
"C:/Program Files/Git/mingw64/bin/make" -s gpg-import GPG_KEY_FILES="secret.gpg"
gpg: directory '/home/application/.gnupg' created
gpg: keybox '/home/application/.gnupg/pubring.kbx' created
gpg: /home/application/.gnupg/trustdb.gpg: trustdb created
gpg: key BBBE654440E720C1: public key "Alice Doe <alice@example.com>" imported
gpg: key BBBE654440E720C1: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
"C:/Program Files/Git/mingw64/bin/make" -s gpg-import GPG_KEY_FILES=".dev/gpg-keys/*"
gpg: key BBBE654440E720C1: "Alice Doe <alice@example.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

Adding (new) team members

Let's start by adding our own user to git-secret

make secret-add-user EMAIL="alice@example.com"

$ make secret-add-user EMAIL="alice@example.com"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="tell alice@example.com"
git-secret: done. alice@example.com added as user(s) who know the secret.

And verify that it worked via

make secret-show-users

$ make secret-show-users
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="whoknows"
alice@example.com

Adding and encrypting files

Let's add a new encrypted file secret_password.txt.

Create the file

echo "my_new_secret_password" > secret_password.txt

Add it to .gitignore

echo "secret_password.txt" >> .gitignore

Add it to git-secret

make secret-add FILE="secret_password.txt"

$ make secret-add FILE="secret_password.txt"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="add secret_password.txt"
git-secret: 1 item(s) added.

Encrypt all files

make secret-encrypt

$ make secret-encrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="hide"
git-secret: done. 1 of 1 files are hidden.

$ ls secret_password.txt.secret
secret_password.txt.secret

Decrypt files

Let's first remove the "plain" secret_password.txt file

rm secret_password.txt

$ rm secret_password.txt

$ ls secret_password.txt
ls: cannot access 'secret_password.txt': No such file or directory

and then decrypt the encrypted one.

make secret-decrypt

$ make secret-decrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f"
git-secret: done. 1 of 1 files are revealed.

$ cat secret_password.txt
my_new_secret_password

Caution: If the secret gpg key is password protected (e.g. 123456), run

make secret-decrypt-with-password GPG_PASSWORD=123456

You could also add the GPG_PASSWORD variable to the .make/.env file as a local default value so that you wouldn't have to specify the value every time and could then simply run

make secret-decrypt-with-password

without passing GPG_PASSWORD

Removing files

Remove the secret_password.txt file we added previously:

make secret-remove FILE="secret_password.txt"

$ make secret-remove FILE="secret_password.txt"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="remove secret_password.txt"
git-secret: removed from index.
git-secret: ensure that files: [secret_password.txt] are now not ignored.

Caution: this will neither remove the secret_password.txt file nor the secret_password.txt.secret file automatically"

$ ls -l | grep secret_password.txt
-rw-r--r-- 1 Pascal 197121     19 Mar 31 14:03 secret_password.txt
-rw-r--r-- 1 Pascal 197121    358 Mar 31 14:02 secret_password.txt.secret

But even though the encrypted secret_password.txt.secret file still exists, it will not be decrypted:

$ make secret-decrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f"
git-secret: done. 0 of 0 files are revealed.

Removing team members

Removing a team member can be done via

make secret-remove-user EMAIL="alice@example.com"

$ make secret-remove-user EMAIL="alice@example.com"
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="killperson alice@example.com"
git-secret: removed keys.
git-secret: now [alice@example.com] do not have an access to the repository.
git-secret: make sure to hide the existing secrets again.

If there are any users left, we must make sure to re-encrypt the secrets via

make secret-encrypt

Otherwise (if no more users are left) git-secret would simply error out

$ make secret-decrypt
"C:/Program Files/Git/mingw64/bin/make" -s git-secret ARGS="reveal -f"
git-secret: abort: no public keys for users found. run 'git secret tell email@address'.
make[1]: *** [.make/01-00-application-setup.mk:57: git-secret] Error 1
make: *** [.make/01-00-application-setup.mk:69: secret-decrypt] Error 2

Caution: Please keep in mind to rotate the secrets themselves as well!

Pros and cons

Pro

very low barrier to entry:
- no third party service required
- easy to integrate in existing codebases, because the secrets are located directly in the codebase
- everything can be handled through docker (no additional local software necessary)
once set up, it is very easy/convenient to use and can be integrated in a team workflow
changes to secrets can be reviewed before they are merged
- this leads to less fuck-ups on deployments
"everything" is in the repository, which brings a lot of familiar benefits like
- version control
- a single git pull is the only thing you need to get everything (=> good dev experience)

Cons

some overhead during onboarding and offboarding
the secret key must be put in the root of the repository at ./secret.gpg
no fine grained permissions for different secrets, e.g. the mysql password on production and staging can not be treated differently
- if somebody can decrypt secrets, ALL of them are exposed
if the a secret key ever gets leaked, all secrets are compromised
- => can be mitigated (to a degree) by using a passphrase on the secret key
- => this is kinda true for any other system that stores secrets as well BUT third parties could probably implement additional measures like multi factor authentication
secrets are versioned alongside the users that have access, i.e. even if a user is removed at some point, he can still decrypt a previous version of the encrypted secrets
- => must be mitigated by rotating the secrets themselves as well

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You are now able to encrypt and decrypt secret files so that they can be stored directly in the git repository.

In the next part of this tutorial, we will set up a CI pipeline for dockerized PHP Apps on Github and Gitlab that decrypts all necessary secrets and then runs our tests and qa tools.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

;

Set up PHP QA tools and control them via make [Tutorial Part 5]

Pascal Landau — Fri, 01 Jul 2022 06:37:57 +0000

This article appeared first on https://www.pascallandau.com/ at Set up PHP QA tools and control them via make [Tutorial Part 5]

In the fifth part of this tutorial series on developing PHP on Docker we will setup some PHP code quality tools and provide a convenient way to control them via GNU make.

FYI: Originally I wanted this tutorial to be a part of
Create a CI pipeline for dockerized PHP Apps
because QA checks are imho vital part of a CI setup - but it kinda grew "too big" and took a way too much space from, well, actually setting up the CI pipelines :)

All code samples are publicly available in my Docker PHP Tutorial repository on Github. You find the branch with the final result of this tutorial at part-5-php-qa-tools-make-docker.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Run Laravel 9 on Docker in 2022 and the following one is Use git-secret to encrypt secrets in the repository.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
- The QA tools
  - phpcs and phpcbf
  - phpstan
  - php-parallel-lint
  - composer-require-checker
  - Additional tools (out of scope)
- QA make targets
  - The qa target
  - The execute "function"
  - Parallel execution and a helper target
  - Sprinkle some color on top
- Further updates in the codebase
- Wrapping up

Introduction

Code quality tools ensure a baseline of code quality by automatically checking certain rules, e.g. code style definitions, proper usage of types, proper declaration of dependencies, etc. When run regularly they are a great way to enforce better code and are thus a
perfect fit for a CI pipeline. For this tutorial, I'm going to setup the following tools:

Style Checker: phpcs
Static Analyzer: phpstan
Code Linter: php-parallel-lint
Dependency Checker: composer-require-checker

and provide convenient access through a qa make target. The end result will look like this:

FYI: When we started out with using code quality tools in general, we have used GrumPHP - and I would still recommend it. We have only transitioned away from it because make gives us a little more flexibility and control.

You can find the "final" makefile at .make/01-02-application-qa.mk.

CAUTION: The Makefile is build on top of the setup that I introduced in Docker from scratch for PHP 8.1 Applications in 2022, so please refer to that tutorial if anything is not clear.

##@ [Application: QA]

# variables
CORES?=$(shell (nproc  || sysctl -n hw.ncpu) 2> /dev/null)

# constants
 ## files
ALL_FILES=./
APP_FILES=app/
TEST_FILES=tests/

 ## bash colors
RED:=\033[0;31m
GREEN:=\033[0;32m
YELLOW:=\033[0;33m
NO_COLOR:=\033[0m

# Tool CLI config
PHPUNIT_CMD=php vendor/bin/phpunit
PHPUNIT_ARGS= -c phpunit.xml
PHPUNIT_FILES=
PHPSTAN_CMD=php vendor/bin/phpstan analyse
PHPSTAN_ARGS=--level=9
PHPSTAN_FILES=$(APP_FILES) $(TEST_FILES)
PHPCS_CMD=php vendor/bin/phpcs
PHPCS_ARGS=--parallel=$(CORES) --standard=psr12
PHPCS_FILES=$(APP_FILES)
PHPCBF_CMD=php vendor/bin/phpcbf
PHPCBF_ARGS=$(PHPCS_ARGS)
PHPCBF_FILES=$(PHPCS_FILES)
PARALLEL_LINT_CMD=php vendor/bin/parallel-lint
PARALLEL_LINT_ARGS=-j 4 --exclude vendor/ --exclude .docker --exclude .git
PARALLEL_LINT_FILES=$(ALL_FILES)
COMPOSER_REQUIRE_CHECKER_CMD=php vendor/bin/composer-require-checker
COMPOSER_REQUIRE_CHECKER_ARGS=--ignore-parse-errors

# call with NO_PROGRESS=true to hide tool progress (makes sense when invoking multiple tools together)
NO_PROGRESS?=false
ifeq ($(NO_PROGRESS),true)
    PHPSTAN_ARGS+= --no-progress
    PARALLEL_LINT_ARGS+= --no-progress
else
    PHPCS_ARGS+= -p
    PHPCBF_ARGS+= -p
endif

# Use NO_PROGRESS=false when running individual tools.
# On  NO_PROGRESS=true  the corresponding tool has no output on success
#                       apart from its runtime but it will still print 
#                       any errors that occured. 
define execute
    if [ "$(NO_PROGRESS)" = "false" ]; then \
        eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)"; \
    else \
        START=$$(date +%s); \
        printf "%-35s" "$@"; \
        if OUTPUT=$$(eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)" 2>&1); then \
            printf " $(GREEN)%-6s$(NO_COLOR)" "done"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
        else \
            printf " $(RED)%-6s$(NO_COLOR)" "fail"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
            echo "$$OUTPUT"; \
            printf "\n"; \
            exit 1; \
        fi; \
    fi
endef

.PHONY: test
test: ## Run all tests
    @$(EXECUTE_IN_APPLICATION_CONTAINER) $(PHPUNIT_CMD) $(PHPUNIT_ARGS) $(ARGS)

.PHONY: phplint
phplint: ## Run phplint on all files
    @$(call execute,$(PARALLEL_LINT_CMD),$(PARALLEL_LINT_ARGS),$(PARALLEL_LINT_FILES), $(ARGS))

.PHONY: phpcs
phpcs: ## Run style check on all application files
    @$(call execute,$(PHPCS_CMD),$(PHPCS_ARGS),$(PHPCS_FILES), $(ARGS))

.PHONY: phpcbf
phpcbf: ## Run style fixer on all application files
    @$(call execute,$(PHPCBF_CMD),$(PHPCBF_ARGS),$(PHPCBF_FILES), $(ARGS))

.PHONY: phpstan
phpstan:  ## Run static analyzer on all application and test files 
    @$(call execute,$(PHPSTAN_CMD),$(PHPSTAN_ARGS),$(PHPSTAN_FILES), $(ARGS))

.PHONY: composer-require-checker
composer-require-checker: ## Run dependency checker
    @$(call execute,$(COMPOSER_REQUIRE_CHECKER_CMD),$(COMPOSER_REQUIRE_CHECKER_ARGS),"", $(ARGS))

.PHONY: qa
qa: ## Run code quality tools on all files
    @"$(MAKE)" -j $(CORES) -k --no-print-directory --output-sync=target qa-exec NO_PROGRESS=true

.PHONY: qa-exec
qa-exec: phpstan \
   phplint \
   composer-require-checker \
   phpcs

The QA tools

phpcs and phpcbf

phpcs is the CLI tool of the style checker squizlabs/PHP_CodeSniffer. It also comes with phpcbf - a tool to automatically fix style errors.

Installation via composer:

make composer ARGS="require --dev squizlabs/php_codesniffer"

For now we will simply use the pre-configured ruleset for PSR-12: Extended Coding Style. When run in the application container for the first time on the app directory via

vendor/bin/phpcs --standard=PSR12 --parallel=4 -p app

i.e.

--standard=PSR12 => use the PSR12 ruleset
--parallel=4     => run with 4 parallel processes
-p               => show the progress

we get the following result:

root:/var/www/app# vendor/bin/phpcs --standard=PSR12 --parallel=4 -p app

FILE: /var/www/app/app/Console/Kernel.php
----------------------------------------------------------------------
FOUND 2 ERRORS AFFECTING 1 LINE
----------------------------------------------------------------------
 28 | ERROR | [x] Expected at least 1 space before "."; 0 found
 28 | ERROR | [x] Expected at least 1 space after "."; 0 found
----------------------------------------------------------------------
PHPCBF CAN FIX THE 2 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------


FILE: /var/www/app/app/Http/Controllers/HomeController.php
----------------------------------------------------------------------
FOUND 4 ERRORS AFFECTING 2 LINES
----------------------------------------------------------------------
 37 | ERROR | [x] Expected at least 1 space before "."; 0 found
 37 | ERROR | [x] Expected at least 1 space after "."; 0 found
 45 | ERROR | [x] Expected at least 1 space before "."; 0 found
 45 | ERROR | [x] Expected at least 1 space after "."; 0 found
----------------------------------------------------------------------
PHPCBF CAN FIX THE 4 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------


FILE: /var/www/app/app/Jobs/InsertInDbJob.php
-------------------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
-------------------------------------------------------------------------------
 13 | ERROR | [x] Each imported trait must have its own "use" import statement
-------------------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
-------------------------------------------------------------------------------


FILE: /var/www/app/app/Models/User.php
-------------------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
-------------------------------------------------------------------------------
 13 | ERROR | [x] Each imported trait must have its own "use" import statement
-------------------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
-------------------------------------------------------------------------------

All errors can be fixed automatically with phpcbf:

root:/var/www/app# vendor/bin/phpcbf --standard=PSR12 --parallel=4 -p app

PHPCBF RESULT SUMMARY
-------------------------------------------------------------------------
FILE                                                     FIXED  REMAINING
-------------------------------------------------------------------------
/var/www/app/app/Console/Kernel.php                      2      0
/var/www/app/app/Http/Controllers/HomeController.php     4      0
/var/www/app/app/Jobs/InsertInDbJob.php                  1      0
/var/www/app/app/Models/User.php                         1      0
-------------------------------------------------------------------------
A TOTAL OF 8 ERRORS WERE FIXED IN 4 FILES
-------------------------------------------------------------------------

Time: 411ms; Memory: 8MB

and a follow-up run of phpcs doesn't show any more errors:

root:/var/www/app# vendor/bin/phpcs --standard=PSR12 --parallel=4 -p app
.................... 20 / 20 (100%)


Time: 289ms; Memory: 8MB

phpstan

phpstan is the CLI tool of the static code analyzer phpstan/phpstan (see also the full PHPStan documentation). It provides some default "levels" of increasing strictness to report potential bugs based on the AST of the analyzed PHP code.

Installation via composer:

make composer ARGS="require --dev phpstan/phpstan"

Since this is a "fresh" codebase with very little code let's go for the highest level 9 (as of 2022-04-24) and run it in the application container on the app and tests directories via:

vendor/bin/phpstan analyse app tests --level=9

--level=9        => use level 9

root:/var/www/app# vendor/bin/phpstan analyse app tests --level=9
 25/25 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

 ------ ------------------------------------------------------------------------------------------------------------------
  Line   app/Commands/SetupDbCommand.php
 ------ ------------------------------------------------------------------------------------------------------------------
  22     Method App\Commands\SetupDbCommand::getOptions() return type has no value type specified in iterable type array.
         � See: https://phpstan.org/blog/solving-phpstan-no-value-type-specified-in-iterable-type
  34     Method App\Commands\SetupDbCommand::handle() has no return type specified.
 ------ ------------------------------------------------------------------------------------------------------------------

 ------ -------------------------------------------------------------------------------------------------------
  Line   app/Http/Controllers/HomeController.php
 ------ -------------------------------------------------------------------------------------------------------
  22     Parameter #1 $jobId of class App\Jobs\InsertInDbJob constructor expects string, mixed given.
  25     Part $jobId (mixed) of encapsed string cannot be cast to string.
  35     Call to an undefined method Illuminate\Redis\Connections\Connection::lRange().
  62     Call to an undefined method Illuminate\Contracts\View\Factory|Illuminate\Contracts\View\View::with().
 ------ -------------------------------------------------------------------------------------------------------

 ------ ------------------------------------------------------------------------------------------------------------------
  Line   app/Http/Middleware/Authenticate.php
 ------ ------------------------------------------------------------------------------------------------------------------
  17     Method App\Http\Middleware\Authenticate::redirectTo() should return string|null but return statement is missing.
 ------ ------------------------------------------------------------------------------------------------------------------

 ------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Line   app/Http/Middleware/RedirectIfAuthenticated.php
 ------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  26     Method App\Http\Middleware\RedirectIfAuthenticated::handle() should return Illuminate\Http\RedirectResponse|Illuminate\Http\Response but returns Illuminate\Http\RedirectResponse|Illuminate\Routing\Redirector.
 ------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 ------ -----------------------------------------------------------------------
  Line   app/Jobs/InsertInDbJob.php
 ------ -----------------------------------------------------------------------
  22     Method App\Jobs\InsertInDbJob::handle() has no return type specified.
 ------ -----------------------------------------------------------------------

 ------ -------------------------------------------------
  Line   app/Providers/RouteServiceProvider.php
 ------ -------------------------------------------------
  36     PHPDoc tag @var above a method has no effect.
  36     PHPDoc tag @var does not specify variable name.
  60     Cannot access property $id on mixed.
 ------ -------------------------------------------------

 ------ ----------------------------------------------------------------------------------------------------------------------------------------------------------
  Line   tests/Feature/App/Http/Controllers/HomeControllerTest.php
 ------ ----------------------------------------------------------------------------------------------------------------------------------------------------------
  24     Method Tests\Feature\App\Http\Controllers\HomeControllerTest::test___invoke() has parameter $params with no value type specified in iterable type array.
         � See: https://phpstan.org/blog/solving-phpstan-no-value-type-specified-in-iterable-type
  38     Method Tests\Feature\App\Http\Controllers\HomeControllerTest::__invoke_dataProvider() return type has no value type specified in iterable type array.
         � See: https://phpstan.org/blog/solving-phpstan-no-value-type-specified-in-iterable-type
 ------ ----------------------------------------------------------------------------------------------------------------------------------------------------------

 ------ ---------------------------------------------------------------------------------------------------------------------
  Line   tests/TestCase.php
 ------ ---------------------------------------------------------------------------------------------------------------------
  68     Cannot access offset 'database' on mixed.
  71     Parameter #1 $config of method Illuminate\Database\Connectors\MySqlConnector::connect() expects array, mixed given.
 ------ ---------------------------------------------------------------------------------------------------------------------


 [ERROR] Found 16 errors

After fixing (or ignoring :P) all errors, we now get

root:/var/www/app# vendor/bin/phpstan analyse app tests --level=9
25/25 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

[OK] No errors

php-parallel-lint

php-parallel-lint is the CLI tool of the PHP code linter php-parallel-lint/PHP-Parallel-Lint. It ensures that all PHP files are syntactically correct.

Installation via composer:

make composer ARGS="require --dev php-parallel-lint/php-parallel-lint"

"Parallel" is already in the name, so we run it on the full codebase ./ with 4 parallel processes and exclude the .git and vendor directories to speed up the execution via

vendor/bin/parallel-lint -j 4 --exclude .git --exclude vendor ./

i.e.

-j 4                              => use 4 parallel processes
--exclude .git --exclude vendor   => ignore the .git/ and vendor/ directories

we get

root:/var/www/app# vendor/bin/parallel-lint -j 4 --exclude .git --exclude vendor ./
PHP 8.1.1 | 4 parallel jobs
............................................................ 60/61 (98 %)
.                                                            61/61 (100 %)


Checked 61 files in 0.2 seconds
No syntax error found

No further TODOs here.

composer-require-checker

composer-require-checker is the CLI tool of the dependency checker maglnet/ComposerRequireChecker. The tool ensures that the composer.json file contains all dependencies that are used in the codebase.

Installation via composer:

make composer ARGS="require --dev maglnet/composer-require-checker"

Run it via

vendor/bin/composer-require-checker check

root:/var/www/app# vendor/bin/composer-require-checker check
ComposerRequireChecker 4.0.0@baa11a4e9e5072117e3d180ef16c07036cafa4a2
The following 1 unknown symbols were found:
+---------------------------------------------+--------------------+
| Unknown Symbol                              | Guessed Dependency |
+---------------------------------------------+--------------------+
| Symfony\Component\Console\Input\InputOption |                    |
+---------------------------------------------+--------------------+

What's going on here?

We use Symfony\Component\Console\Input\InputOption in our \App\Commands\SetupDbCommand and the code doesn't "fail" because InputOption is defined in thesymfony/console package that is a
transitive dependency of laravel/framework, see the laravel/framework composer.json file.

I.e. the symfony/console package does actually exist in our vendor directory - but since we also use it as a first-party-dependency directly in our code, we must declare the dependency explicitly. Otherwise, Laravel might at some point decide to drop symfony/console and we would be left with broken code.

To fix this, I run

make composer ARGS="require symfony/console"

which will update the composer.json file and add the dependency. Running composer-require-checker again will now yield no further errors.

root:/var/www/app# vendor/bin/composer-require-checker check
ComposerRequireChecker 4.0.0@baa11a4e9e5072117e3d180ef16c07036cafa4a2
There were no unknown symbols found.

Additional tools (out of scope)

In general, I'm a huge fan of code quality tools and we use them quite extensively. At some point I'll probably dedicate a whole article to go over them in detail - but for now I'm just gonna leave a list for inspiration:

brianium/paratest
- Running PhpUnit tests in parallel
- malukenho/mcbumpface
- Update the versions in the composer.json file after an update
- qossmic/deptrac-shim
- A shim for qossmic/deptrac: A tool to define dependency layers based on e.g. namespaces
- icanhazstring/composer-unused
- Show dependencies in the composer.json that are not used in the codebase
- roave/security-advisories
- Gives a warning when packages with known vulnerabilities are used
- Alternative: local-php-security-checker

QA make targets

You might have noticed that all tools have their own configuration options. Instead of remembering each of them, I'll define corresponding make targets in .make/01-02-application-qa.mk. The easiest way to do so would be to "hard-code" the exact commands that I ran previously, e.g.

.PHONY: phpstan
phpstan:  ## Run static analyzer on all application and test files 
    @$(EXECUTE_IN_APPLICATION_CONTAINER) vendor/bin/phpstan analyse app tests --level=9

(Please refer to the Run commands in the docker containers section in the previous tutorial for an explanation of the EXECUTE_IN_APPLICATION_CONTAINER variable).

However, this implementation is quite inflexible: What if we want to check a single file or try out other options? So let's create some variables instead:

PHPSTAN_CMD=php vendor/bin/phpstan analyse
PHPSTAN_ARGS=--level=9
PHPSTAN_FILES=$(APP_FILES) $(TEST_FILES)

.PHONY: phpstan
phpstan: ## Run static analyzer on all application and test files 
    @$(EXECUTE_IN_APPLICATION_CONTAINER) $(PHPSTAN_CMD) $(PHPSTAN_ARGS) $(PHPSTAN_FILES)

This target allows me to override the defaults and e.g. check only the file app/Commands/SetupDbCommand.php with --level=1

make phpstan PHPSTAN_FILES=app/Commands/SetupDbCommand.php PHPSTAN_ARGS="--level=1"

$ make phpstan PHPSTAN_FILES=app/Commands/SetupDbCommand.php PHPSTAN_ARGS="--level=1" 
 1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%


 [OK] No errors

The remaining tool variables can be configured in the exact same way:

# constants
 ## files
ALL_FILES=./
APP_FILES=app/
TEST_FILES=tests/

# Tool CLI config
PHPUNIT_CMD=php vendor/bin/phpunit
PHPUNIT_ARGS= -c phpunit.xml
PHPUNIT_FILES=
PHPSTAN_CMD=php vendor/bin/phpstan analyse
PHPSTAN_ARGS=--level=9
PHPSTAN_FILES=$(APP_FILES) $(TEST_FILES)
PHPCS_CMD=php vendor/bin/phpcs
PHPCS_ARGS=--parallel=$(CORES) --standard=psr12
PHPCS_FILES=$(APP_FILES)
PHPCBF_CMD=php vendor/bin/phpcbf
PHPCBF_ARGS=$(PHPCS_ARGS)
PHPCBF_FILES=$(PHPCS_FILES)
PARALLEL_LINT_CMD=php vendor/bin/parallel-lint
PARALLEL_LINT_ARGS=-j 4 --exclude vendor/ --exclude .docker --exclude .git
PARALLEL_LINT_FILES=$(ALL_FILES)
COMPOSER_REQUIRE_CHECKER_CMD=php vendor/bin/composer-require-checker
COMPOSER_REQUIRE_CHECKER_ARGS=--ignore-parse-errors

The `qa` target

From a workflow perspective I usually want to run all configured qa tools instead of each one individually (being able to run individually is still great if a tool fails, though).

A trivial approach would be a new target that uses all individual tool targets as preconditions:

.PHONY: qa
qa: phpstan \
   phplint \
   composer-require-checker \
   phpcs

But we can do better, because this target produces quite a noisy output:

$ make qa
 25/25 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%


 [OK] No errors

PHP 8.1.1 | 4 parallel jobs
............................................................ 60/61 (98 %)
.                                                            61/61 (100 %)


Checked 61 files in 0.3 seconds
No syntax error found
ComposerRequireChecker 4.0.0@baa11a4e9e5072117e3d180ef16c07036cafa4a2
There were no unknown symbols found.
.................... 20 / 20 (100%)


Time: 576ms; Memory: 8MB

I'd rather have something like this:

$ make qa
phplint                             done   took 1s
phpcs                               done   took 1s
phpstan                             done   took 3s
composer-require-checker            done   took 6s

The `execute` "function"

We'll make this work by suppressing the tool output and using a user-defined execute make function to format all targets nicely.

Though "function" isn't quite correct here, because it's rather a multiline variable defined via define ... endef that is then "invoked" via the call function.

# File: 01-02-application-qa.mk

# call with NO_PROGRESS=true to hide tool progress (makes sense when invoking multiple tools together)
NO_PROGRESS?=false
ifeq ($(NO_PROGRESS),true)
    PHPSTAN_ARGS+= --no-progress
endif


# Use NO_PROGRESS=false when running individual tools.
# On  NO_PROGRESS=true  the corresponding tool has no output on success
#                       apart from its runtime but it will still print 
#                       any errors that occured. 
define execute
    if [ "$(NO_PROGRESS)" = "false" ]; then \
        eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)"; \
    else \
        START=$$(date +%s); \
        printf "%-35s" "$@"; \
        if OUTPUT=$$(eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)" 2>&1); then \
            printf " %-6s" "done"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $${RUNTIME}s\n"; \
        else \
            printf " %-6s" "fail"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $${RUNTIME}s\n"; \
            echo "$$OUTPUT"; \
            printf "\n"; \
            exit 1; \
        fi; \
    fi
endef

the NO_PROGRESS variable is set to false by default and will cause a target to be invoked as before, showing all its output immediately
- if the variable is set to true, the target is instead invoked via eval and the output is captured in the OUTPUT bash variable that will only be printed if the result of the invocation is faulty

The tool targets are then adjusted to use the new function.

.PHONY: phpstan
phpstan: ## Run static analyzer on all application and test files
    @$(call execute,$(PHPSTAN_CMD),$(PHPSTAN_ARGS),$(PHPSTAN_FILES),$(ARGS))

We can now call the phpstan target with NO_PROGRESS=true like so:

$ make phpstan NO_PROGRESS=true
phpstan                             done   took 4s

An "error" would look likes this:

$ make phpstan NO_PROGRESS=true
phpstan                             fail   took 9s
 ------ ----------------------------------------
  Line   app/Providers/RouteServiceProvider.php
 ------ ----------------------------------------
  49     Cannot access property $id on mixed.
 ------ ----------------------------------------

Parallel execution and a helper target

Technically, this also already works with the qa target and we can even speed up the process by
running the tools in parallel with the -j flag for "Parallel Execution"

$ make -j 4 qa NO_PROGRESS=true
phpstan                            phplint                            composer-require-checker           phpcs                               done   took 5s
done   took 5s
done   took 7s
done   took 10s

Well... not quite what we wanted. We also need to use --output-sync=target to controll the "Output During Parallel Execution"

$ make -j 4 --output-sync=target qa NO_PROGRESS=true
phpstan                             done   took 3s
phplint                             done   took 1s
composer-require-checker            done   took 5s
phpcs                               done   took 1s

Since this is quite a mouthful to type, we'll use a helper target qa-exec for running the tools and put all the inconvenient-to-type options in the final qa target.

# File: 01-02-application-qa.mk
#...

# variables
CORES?=$(shell (nproc  || sysctl -n hw.ncpu) 2> /dev/null)

.PHONY: qa
qa: ## Run code quality tools on all files
    @"$(MAKE)" -j $(CORES) -k --no-print-directory --output-sync=target qa-exec NO_PROGRESS=true

.PHONY: qa-exec
qa-exec: phpstan \
   phplint \
   composer-require-checker \
   phpcs

For the number of parallel processes I use nproc (works on Linux and Windows) resp. sysctl -n hw.ncpu (works on Mac) to determine the number of available cores. If you dedicate less resources to docker you might want to hard-code this setting to a lower value (e.g. by adding a CORES variable in the .make/.env file).

Sprinkle some color on top

The final piece for getting to the output mentioned in the Introduction is the bash-coloring:

To make this work, we need to understand first how colors work in bash:

This [coloring] can be accomplished by adding a \e [or \033] at the beginning to form an
escape sequence. The escape sequence for specifying color codes is \e[COLORm
(COLOR represents our (numeric) color code in this case).

(via Adding colors to Bash scripts)

E.g. the following script will print a green text:

printf "\033[0;32mThis text is green\033[0m"

So we define the required colors as variables and use them in the corresponding places in the execute function:

 ## bash colors
RED:=\033[0;31m
GREEN:=\033[0;32m
YELLOW:=\033[0;33m
NO_COLOR:=\033[0m

# ...

# Use NO_PROGRESS=false when running individual tools.
# On  NO_PROGRESS=true  the corresponding tool has no output on success
#                       apart from its runtime but it will still print 
#                       any errors that occured. 
define execute
    if [ "$(NO_PROGRESS)" = "false" ]; then \
        eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)"; \
    else \
        START=$$(date +%s); \
        printf "%-35s" "$@"; \
        if OUTPUT=$$(eval "$(EXECUTE_IN_APPLICATION_CONTAINER) $(1) $(2) $(3) $(4)" 2>&1); then \
            printf " $(GREEN)%-6s$(NO_COLOR)" "done"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
        else \
            printf " $(RED)%-6s$(NO_COLOR)" "fail"; \
            END=$$(date +%s); \
            RUNTIME=$$((END-START)) ;\
            printf " took $(YELLOW)$${RUNTIME}s$(NO_COLOR)\n"; \
            echo "$$OUTPUT"; \
            printf "\n"; \
            exit 1; \
        fi; \
    fi
endef

Please note, that i did not include the tests in the qa target. I like to run those separately, because our tests usually take a lot longer to execute. So in my day-to-day work I would run make qa and make test to ensure that code quality and tests are passing:

$ make qa
phplint                             done   took 1s
phpcs                               done   took 1s
composer-require-checker            done   took 14s
phpstan                             done   took 16s

$ make test
PHPUnit 9.5.19 #StandWithUkraine

.......                                                             7 / 7 (100%)

Time: 00:03.772, Memory: 28.00 MB

OK (7 tests, 13 assertions)

Further updates in the codebase

I've also cleaned up the codebase a little in branch part-5-php-qa-tools-make-docker and even though those changes have nothing to todo with "QA tools" I didn't want to leave them unnoticed:

removing unnecessary files (.styleci.yml, package.json, webpack.mix.js)
- removing unused values from the .env.example file
- run a composer update to get the latest Laravel version
- add a show-help script to the scripts section of the composer.json file that references the Makefile (see also this discussion on Twitter)
```
{
    "scripts": {
        "show-help": [
            "make"
        ]
    },
    "scripts-descriptions": {
        "show-help": "Display available 'make' commands (we use make instead of composer scripts)."
    }
}
```
- replace docker-compose with docker compose to use compose v2

For some reason, the last point caused some trouble because Linux and Docker Desktop for Windows require a -T flag for the exec command to disable a TTY allocation in some cases. Whereas on Docker Desktop for Mac the missing TTY lead to a cluttered output ("staircase effect").

Thus I modified the Makefile to populate a DOCKER_COMPOSE_EXEC_OPTIONS variable based on the OS

# Add the -T options to "docker compose exec" to avoid the 
# "panic: the handle is invalid"
# error on Windows and Linux 
# @see https://stackoverflow.com/a/70856332/413531
DOCKER_COMPOSE_EXEC_OPTIONS=-T

# OS is defined for WIN systems, so "uname" will not be executed
OS?=$(shell uname)
ifeq ($(OS),Windows_NT)
    # Windows requires the .exe extension, otherwise the entry is ignored
 # @see https://stackoverflow.com/a/60318554/413531
 SHELL := bash.exe
else ifeq ($(OS),Darwin)
    # On Mac, the -T must be omitted to avoid cluttered output
 # @see https://github.com/moby/moby/issues/37366#issuecomment-401157643
 DOCKER_COMPOSE_EXEC_OPTIONS=
endif

And use the variable when defining EXECUTE_IN_*_CONTAINER in .make/02-00-docker.mk

ifeq ($(EXECUTE_IN_CONTAINER),true)
    EXECUTE_IN_ANY_CONTAINER:=$(DOCKER_COMPOSE) exec $(DOCKER_COMPOSE_EXEC_OPTIONS) --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME)
    EXECUTE_IN_APPLICATION_CONTAINER:=$(DOCKER_COMPOSE) exec $(DOCKER_COMPOSE_EXEC_OPTIONS) --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME_APPLICATION)
    EXECUTE_IN_WORKER_CONTAINER:=$(DOCKER_COMPOSE) exec $(DOCKER_COMPOSE_EXEC_OPTIONS) --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME_PHP_WORKER)
endif

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. You should now have a blueprint for adding code quality tools for your dockerized application and way to conveniently control them through a Makefile.

In the next part of this tutorial, we will set up git secret to encrypt secret values and store them directly in the git repository.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

Run Laravel 9 on Docker in 2022 [Tutorial Part 4]

Pascal Landau — Tue, 28 Jun 2022 07:35:36 +0000

This article appeared first on https://www.pascallandau.com/ at Run Laravel 9 on Docker in 2022 [Tutorial Part 4]

In this third subpart of the fourth part of this tutorial series on developing PHP on Docker we will install Laravel and make sure our setup works for Artisan Commands, a Redis Queue and Controllers
for the front end requests.

All code samples are publicly available in my Docker PHP Tutorial repository on Github. You find the branch for this tutorial at part-4-3-run-laravel-9-docker-in-2022.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was PhpStorm, Docker and Xdebug 3 on PHP 8.1 in 2022 and the following one is Set up PHP QA tools and control them via make.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
Install extensions
Install Laravel
Update the PHP POC
- config
  - database connection
  - queue connection
- Controllers
- Commands
- Jobs and workers
- Tests
Makefile updates
- Clearing the queue
Running the POC
Wrapping up

Introduction

The goal of this tutorial is to run the PHP POC from part 4.1 using Laravel as a framework instead of "plain PHP". We'll use the newest version of Laravel (Laravel 9) that was released at the beginning of February 2022.

Install extensions

Before Laravel can be installed, we need to add the necessary extensions of the framework (and all its dependencies) to the php-base image:

# File: .docker/images/php/base/Dockerfile

# ...

RUN apk add --update --no-cache  \
        php-curl~=${TARGET_PHP_VERSION} \

Install Laravel

We'll start by creating a new Laravel project with composer

composer create-project --prefer-dist laravel/laravel /tmp/laravel "9.*" --no-install --no-scripts

The files are added to /tmp/laravel because composer projects cannot be created in non-empty folders , so we need to create the project in a temporary location first and move it afterwards.

Since I don't have PHP 8 installed on my laptop, I'll execute the command in the application docker container via

make execute-in-container DOCKER_SERVICE_NAME="application" COMMAND='composer create-project --prefer-dist laravel/laravel /tmp/laravel "9.*" --no-install --no-scripts'

and then move the files into the application directory via

rm -rf public/ tests/ composer.* phpunit.xml
make execute-in-container DOCKER_SERVICE_NAME="application" COMMAND="bash -c 'mv -n /tmp/laravel/{.*,*} .' && rm -f /tmp/laravel"
cp .env.example .env

Notes

composer install is skipped via --no-install to avoid having to copy over the vendor/ folder (which is super slow on Docker Desktop)
existing directories cannot be overwritten by mv thus I remove public/ and tests/ upfront (as well as the composer and phpunit config files)
mv uses the -n flag so that existing files like our .editorconfig are not overwritten
I need to use bash -c to run the command in the container because otherwise the * wildcard would have no effect in the container

To finalize the installation I need to install the composer dependencies and execute the create-project scripts defined in composer.json:

make composer ARGS=install
make composer ARGS="run-script post-create-project-cmd"

Since our nginx configuration was already pointing to the public/ directory, we can immediately open http://127.0.0.1 in the browser and should see the frontpage of a fresh Laravel installation.

Update the PHP POC

config

We need to update the connection information for the database and the queue (previously configured via dependencies.php) in the .env file

database connection

DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=application_db
DB_USERNAME=root
DB_PASSWORD=secret_mysql_root_password

queue connection

QUEUE_CONNECTION=redis

REDIS_HOST=redis
REDIS_PASSWORD=secret_redis_password

Controllers

The functionality of the previous public/index.php file now lives in the HomeController at app/Http/Controllers/HomeController.php

class HomeController extends Controller
{
    use DispatchesJobs;

    public function __invoke(Request $request, QueueManager $queueManager, DatabaseManager $databaseManager): View
    {
        $jobId = $request->input("dispatch") ?? null;
        if ($jobId !== null) {
            $job = new InsertInDbJob($jobId);
            $this->dispatch($job);

            return $this->getView("Adding item '$jobId' to queue");
        }

        if ($request->has("queue")) {

            /**
             * @var RedisQueue $redisQueue
             */
            $redisQueue = $queueManager->connection();
            $redis =  $redisQueue->getRedis()->connection();
            $queueItems = $redis->lRange("queues:default", 0, 99999);

            $content = "Items in queue\n".var_export($queueItems, true);

            return $this->getView($content);
        }

        if ($request->has("db")) {
            $items = $databaseManager->select($databaseManager->raw("SELECT * FROM jobs"));

            $content = "Items in db\n".var_export($items, true);

            return $this->getView($content);
        }
        $content = <<<HTML
            <ul>
                <li><a href="?dispatch=foo">Dispatch job 'foo' to the queue.</a></li>
                <li><a href="?queue">Show the queue.</a></li>
                <li><a href="?db">Show the DB.</a></li>
            </ul>
            HTML;

        return $this->getView($content);
    }

    private function getView(string $content): View
    {
        return view('home')->with(["content" => $content]);
    }
}

Its content is displayed via the home view located at resources/views/home.blade.php:

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    {!! $content !!}
    </body>
</html>

The controller is added as a route in routes/web.php:

Route::get('/', \App\Http\Controllers\HomeController::class)->name("home");

Commands

We will replace the setup.php script with a SetupDbCommand that is located at app/Commands/SetupDbCommand.php

class SetupDbCommand extends Command
{
    /**
     * @var string
     */
    protected $name = "app:setup-db";

    /**
     * @var string
     */
    protected $description = "Run the application database setup";

    protected function getOptions(): array
    {
        return [
            [
                "drop",
                null,
                InputOption::VALUE_NONE,
                "If given, the existing database tables are dropped and recreated.",
            ],
        ];
    }

    public function handle()
    {
        $drop = $this->option("drop");
        if ($drop) {
            $this->info("Dropping all database tables...");

            $this->call(WipeCommand::class);

            $this->info("Done.");
        }

        $this->info("Running database migrations...");

        $this->call(MigrateCommand::class);

        $this->info("Done.");
    }
}

    public function register()
    {
        $this->commands([
            \App\Commands\SetupDbCommand::class
        ]);
    }

and update the setup-db target in .make/01-00-application-setup.mk to run the artisan Command

.PHONY: setup-db
setup-db: ## Setup the DB tables
    $(EXECUTE_IN_APPLICATION_CONTAINER) php artisan app:setup-db $(ARGS);

We will also create a migration for the jobs table in database/migrations/2022_02_10_000000_create_jobs_table.php:

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('jobs', function (Blueprint $table) {
            $table->id();
            $table->string('value');
        });
    }
};

Jobs and workers

We will replace the worker.php script with InsertInDbJob located at app/Jobs/InsertInDbJob.php

class InsertInDbJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable;

    public function __construct(
        public readonly string $jobId
    ) {
    }

    public function handle(DatabaseManager $databaseManager)
    {
        $databaseManager->insert("INSERT INTO `jobs`(value) VALUES(?)", [$this->jobId]);
    }
}

though this will "only" handle the insertion part. For the worker itself we will use the native \Illuminate\Queue\Console\WorkCommand via

php artisan queue:work

We need to adjust the .docker/images/php/worker/Dockerfile and change

ARG PHP_WORKER_COMMAND="php $APP_CODE_PATH/worker.php"

ARG PHP_WORKER_COMMAND="php $APP_CODE_PATH/artisan queue:work"

Since this change takes place directly in the Dockerfile, we must now rebuild the image

$ make docker-build-image DOCKER_SERVICE_NAME=php-worker

and restart it

$ make docker-up

Tests

I'd also like to take this opportunity to add a Feature test for the HomeController at tests/Feature/App/Http/Controllers/HomeControllerTest.php:

class HomeControllerTest extends TestCase
{
    public function setUp(): void
    {
        parent::setUp();

        $this->setupDatabase();
        $this->setupQueue();
    }

    /**
     * @dataProvider __invoke_dataProvider
     */
    public function test___invoke(array $params, string $expected): void
    {
        $urlGenerator = $this->getDependency(UrlGenerator::class);

        $url = $urlGenerator->route("home", $params);

        $response = $this->get($url);

        $response
            ->assertStatus(200)
            ->assertSee($expected, false)
        ;
    }

    public function __invoke_dataProvider(): array
    {
        return [
            "default"           => [
                "params"   => [],
                "expected" => <<<TEXT
                        <li><a href="?dispatch=foo">Dispatch job 'foo' to the queue.</a></li>
                        <li><a href="?queue">Show the queue.</a></li>
                        <li><a href="?db">Show the DB.</a></li>
                    TEXT
                ,
            ],
            "database is empty" => [
                "params"   => ["db"],
                "expected" => <<<TEXT
                        Items in db
                    array (
                    )
                    TEXT
                ,
            ],
            "queue is empty"    => [
                "params"   => ["queue"],
                "expected" => <<<TEXT
                        Items in queue
                    array (
                    )
                    TEXT
                ,
            ],
        ];
    }

    public function test_shows_existing_items_in_database(): void
    {
        $databaseManager = $this->getDependency(DatabaseManager::class);

        $databaseManager->insert("INSERT INTO `jobs` (id, value) VALUES(1, 'foo');");

        $urlGenerator = $this->getDependency(UrlGenerator::class);

        $params = ["db"];
        $url    = $urlGenerator->route("home", $params);

        $response = $this->get($url);

        $expected = <<<TEXT
                Items in db
            array (
              0 => 
              (object) array(
                 'id' => 1,
                 'value' => 'foo',
              ),
            )
            TEXT;

        $response
            ->assertStatus(200)
            ->assertSee($expected, false)
        ;
    }

    public function test_shows_existing_items_in_queue(): void
    {
        $queueManager = $this->getDependency(QueueManager::class);

        $job = new InsertInDbJob("foo");
        $queueManager->push($job);

        $urlGenerator = $this->getDependency(UrlGenerator::class);

        $params = ["queue"];
        $url    = $urlGenerator->route("home", $params);

        $response = $this->get($url);

        $expectedJobsCount = <<<TEXT
                Items in queue
            array (
              0 => '{
            TEXT;

        $expected = <<<TEXT
            \\\\"jobId\\\\";s:3:\\\\"foo\\\\";
            TEXT;

        $response
            ->assertStatus(200)
            ->assertSee($expectedJobsCount, false)
            ->assertSee($expected, false)
        ;
    }
}

The test checks the database as well as the queue and uses the helper methods $this->setupDatabase() and $this->setupQueue() that I defined in the base test case at tests/TestCase.php as follows

   /**
     * @template T
     * @param class-string<T> $className
     * @return T
     */
    protected function getDependency(string $className)
    {
        return $this->app->get($className);
    }

    protected function setupDatabase(): void
    {
        $databaseManager = $this->getDependency(DatabaseManager::class);

        $actualConnection  = $databaseManager->getDefaultConnection();
        $testingConnection = "testing";
        if ($actualConnection !== $testingConnection) {
            throw new RuntimeException("Database tests are only allowed to run on default connection '$testingConnection'. The current default connection is '$actualConnection'.");
        }

        $this->ensureDatabaseExists($databaseManager);

        $this->artisan(SetupDbCommand::class, ["--drop" => true]);
    }

    protected function setupQueue(): void
    {
        $queueManager = $this->getDependency(QueueManager::class);

        $actualDriver  = $queueManager->getDefaultDriver();
        $testingDriver = "testing";
        if ($actualDriver !== $testingDriver) {
            throw new RuntimeException("Queue tests are only allowed to run on default driver '$testingDriver'. The current default driver is '$actualDriver'.");
        }

        $this->artisan(ClearCommand::class);
    }

    protected function ensureDatabaseExists(DatabaseManager $databaseManager): void
    {
        $connection = $databaseManager->connection();

        try {
            $connection->getPdo();
        } catch (PDOException $e) {
            // e.g. SQLSTATE[HY000] [1049] Unknown database 'testing'
            if ($e->getCode() !== 1049) {
                throw $e;
            }
            $config             = $connection->getConfig();
            $config["database"] = "";

            $connector = new MySqlConnector();
            $pdo       = $connector->connect($config);
            $database  = $connection->getDatabaseName();
            $pdo->exec("CREATE DATABASE IF NOT EXISTS `{$database}`;");
        }
    }

The methods ensure that the tests are only executed if the proper database connection and queue driver is configured. This is done through environment variables and I like using a dedicated .env file located at .env.testing for all testing ENV values instead of defining them in the phpunit.xml config file via <env> elements:

# File: .env.testing

DB_CONNECTION=testing
DB_DATABASE=testing
QUEUE_CONNECTION=testing
REDIS_DB=1000

The corresponding connections have to be configured in the config files

# File: config/database.php

return [
// ...
    'connections' => [
// ...
        'testing' => [
            'driver' => 'mysql',
            'url' => env('DATABASE_URL'),
            'host' => env('DB_HOST'),
            'port' => env('DB_PORT', '3306'),
            'database' => env('DB_DATABASE', 'testing'),
            'username' => env('DB_USERNAME'),
            'password' => env('DB_PASSWORD', ''),
            'unix_socket' => env('DB_SOCKET', ''),
            'charset' => 'utf8mb4',
            'collation' => 'utf8mb4_unicode_ci',
            'prefix' => '',
            'prefix_indexes' => true,
            'strict' => true,
            'engine' => null,
            'options' => extension_loaded('pdo_mysql') ? array_filter([
                PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'),
            ]) : [],
        ],
    ],
// ...
    'redis' => [
// ...
        'testing' => [
            'url' => env('REDIS_URL'),
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD'),
            'port' => env('REDIS_PORT', '6379'),
            'database' => env('REDIS_DB', '1000'),
        ],
    ],
];

# File: config/queue.php

return [
// ...

    'connections' => [
// ...
        'testing' => [
            'driver' => 'redis',
            'connection' => 'testing', // => refers to the "database.redis.testing" config entry
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 90,
            'block_for' => null,
            'after_commit' => false,
        ],
    ],
];

The tests can be executed via make test

$ make test
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker vendor/bin/phpunit -c phpunit.xml
PHPUnit 9.5.13 by Sebastian Bergmann and contributors.

.......                                                             7 / 7 (100%)

Time: 00:02.709, Memory: 28.00 MB

OK (7 tests, 13 assertions)

Makefile updates

Clearing the queue

For convenience while testing I added a make target to clear all items from the queue in .make/01-01-application-commands.mk

.PHONY: clear-queue
clear-queue: ## Clear the job queue
    $(EXECUTE_IN_APPLICATION_CONTAINER) php artisan queue:clear $(ARGS)

Running the POC

Since the POC only uses make targets and we basically just "refactored" them, there is no modification necessary to make the existing test.sh work:

$ bash test.sh


  Building the docker setup


//...


  Starting the docker setup


//...


  Clearing DB


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application application php artisan app:setup-db --drop;
Dropping all database tables...
Dropped all tables successfully.
Done.
Running database migrations...
Migration table created successfully.
Migrating: 2014_10_12_000000_create_users_table
Migrated:  2014_10_12_000000_create_users_table (64.04ms)
Migrating: 2014_10_12_100000_create_password_resets_table
Migrated:  2014_10_12_100000_create_password_resets_table (50.06ms)
Migrating: 2019_08_19_000000_create_failed_jobs_table
Migrated:  2019_08_19_000000_create_failed_jobs_table (58.61ms)
Migrating: 2019_12_14_000001_create_personal_access_tokens_table
Migrated:  2019_12_14_000001_create_personal_access_tokens_table (94.03ms)
Migrating: 2022_02_10_000000_create_jobs_table
Migrated:  2022_02_10_000000_create_jobs_table (31.85ms)
Done.


  Stopping workers


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker supervisorctl stop worker:*;
worker:worker_00: stopped
worker:worker_01: stopped
worker:worker_02: stopped
worker:worker_03: stopped


  Ensuring that queue and db are empty


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in queue
array (
)
    </body>
</html>
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in db
array (
)
    </body>
</html>


  Dispatching a job 'foo'


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Adding item 'foo' to queue
    </body>
</html>


  Asserting the job 'foo' is on the queue


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in queue
array (
  0 => '{"uuid":"7ea63590-2a86-4739-abf8-8a059d41bd60","displayName":"App\\\\Jobs\\\\InsertInDbJob","job":"Illuminate\\\\Queue\\\\CallQueuedHandler@call","maxTries":null,"maxExceptions":null,"failOnTimeout":false,"backoff":null,"timeout":null,"retryUntil":null,"data":{"commandName":"App\\\\Jobs\\\\InsertInDbJob","command":"O:22:\\"App\\\\Jobs\\\\InsertInDbJob\\":11:{s:5:\\"jobId\\";s:3:\\"foo\\";s:3:\\"job\\";N;s:10:\\"connection\\";N;s:5:\\"queue\\";N;s:15:\\"chainConnection\\";N;s:10:\\"chainQueue\\";N;s:19:\\"chainCatchCallbacks\\";N;s:5:\\"delay\\";N;s:11:\\"afterCommit\\";N;s:10:\\"middleware\\";a:0:{}s:7:\\"chained\\";a:0:{}}"},"id":"I3k5PNyGZc6Z5XWCC4gt0qtSdqUZ84FU","attempts":0}',
)
    </body>
</html>


  Starting the workers


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker supervisorctl start worker:*;
worker:worker_00: started
worker:worker_01: started
worker:worker_02: started
worker:worker_03: started


  Asserting the queue is now empty


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in queue
array (
)
    </body>
</html>


  Asserting the db now contains the job 'foo'


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
    </head>
    <body>
    Items in db
array (
  0 =>
  (object) array(
     'id' => 1,
     'value' => 'foo',
  ),
)
    </body>
</html>

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. Laravel 9 should now be up and running on the previously set up docker infrastructure.

In the next part of this tutorial, we will Set up PHP QA tools and control them via make.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

How to build a Docker development setup for PHP Projects [Tutorial Part 1]

Pascal Landau — Mon, 27 Jun 2022 07:46:58 +0000

This article appeared first on https://www.pascallandau.com/ at How to build a Docker development setup for PHP Projects [Tutorial Part 1]

Caution There is a follow-up version of this article available that was published in 2022 - please make sure to read that as well:
Docker from scratch for PHP 8.1 Applications in 2022

In this part of my tutorial series on developing PHP on Docker we'll lay the fundamentals to build a complete development infrastructure and explain how to "structure" the Docker setup as part of a PHP project. Structure as in

folder structure ("what to put where")
Dockerfile templates
solving common problems (file permissions, runtime configuration, ...)

We will also create a minimal container setup consisting of php-fpm, nginx and a workspace container that we refactor from the previous parts of this tutorial.

All code samples are publicly available in my Docker PHP Tutorial repository on Github. The branch for this tutorial is part_3_structuring-the-docker-setup-for-php-projects.

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Setting up PhpStorm with Xdebug for local development on Docker and the following one is Docker from scratch for PHP 8.1 Applications in 2022.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
Structuring the repository
- The .docker folder
- The .shared folder
- docker-test.sh
- .env.example and docker-compose.yml
- The Makefile
Defining services: php-fpm, nginx and workspace
- php-fpm
  - Modifying the pool configuration
  - Custom ENTRYPOINT
- nginx
- workspace (formerly php-cli)
Setting up docker-compose
- docker-compose.yml
- .env.example
- Building and running the containers
- Testing if everything works
Makefile and .bashrc
- Using make as central entry point
- Install make on Windows (MinGW)
- Easy container access via din .bashrc helper
Fundamentals on building the containers
- Understanding build context
- Dockerfile template
- Setting the timezone
- Synchronizing file and folder ownership on shared volumes
- Modifying configuration files
  - Providing additional config files
  - Changing non-static values
- Installing php extensions
- Installing common software
- Cleaning up
- Using ENTRYPOINT for pre-run configuration
  - Providing host.docker.internal for linux host systems
Wrapping up

Introduction

When I started my current role as Head of Marketing Technology at ABOUT YOU back in 2016, we heavily relied on Vagrant (namely: Homestead) as our development infrastructure. Though that was much better than working on our local machines, we've run into a couple of problems along the way (e.g. diverging software, bloated images, slow starting times, complicated readme for onboarding, upgrading php, ...).

Today, everything that we need for the infrastructure is under source control and committed in the same repository that we use for our main application. In effect we get the same infrastructure for every developer including automatic updates "for free". It is extremely easy to tinker around with updates / new tools due to the ephemeral nature of docker as tear down and rebuild only take one command and a couple of minutes.

To get a feeling for how the process feels like, simply execute the following commands.

git clone https://github.com/paslandau/docker-php-tutorial.git
cd docker-php-tutorial
git checkout part_3_structuring-the-docker-setup-for-php-projects
make docker-clean
make docker-init
make docker-build-from-scratch
make docker-test

You should now have a running docker environment to develop PHP on docker (unless something is blocking your port 80/443 or you don't have make installed
;))

Structuring the repository

While playing around with docker I've tried different ways to "structure" files and folders and ended up with the following concepts:

everything related to docker is placed in a .docker directory on on the same level as the main application
in this directory
- each service gets its own subdirectory for configuration
- is a .shared folder containing scripts and configuration required by multiple services
- is an .env.example file containing variables for the docker-compose.yml
- is a docker-test.sh file containing high level tests to validate the docker containers
a Makefile with common instructions to control Docker is placed in the repository root

The result looks roughly like this:

<project>/
├── .docker/
|   ├── .shared/
|   |   ├── config/
|   |   └── scripts/
|   ├── php-fpm/
|   |   └── Dockerfile
|   ├── ... <additional services>/
|   ├── .env.example
|   ├── docker-compose.yml
|   └── docker-test.sh
├── Makefile
├── index.php
└──  ... <additional app files>/

The .docker folder

As I mentioned, for me it makes a lot of sense to keep the infrastructure definition close to the codebase, because it is immediately available to every developer. For bigger projects with multiple components there will be a code-infrastructure-coupling anyways (e.g. in my experience it is usually not possible to simply switch MySQL for PostgreSQL without any other changes) and for a library it is a very convenient (although opinionated) way to get started.

I personally find it rather frustrating when I want to contribute to an open source project but find myself spending a significant amount of time setting the environment up correctly instead of being able to just work on the code.

Ymmv, though (e.g. because you don't want everybody with write access to your app repo also to be able to change your infrastructure code). We actually went a different route previously and had a second repository ("-inf") that would contain the contents of the .docker folder:

<project-inf>/
├── .shared/
|   ├── config/
|   └── scripts/
├── php-fpm/
|   └── Dockerfile
├── ... <additional services>/
├── .env.example
└──  docker-compose.yml

<project>/
├── index.php
└──  ... <additional app files>/

Worked as well, but we often ran into situations where the contents of the repo would be stale for some devs, plus it was simply additional overhead with not other benefits to us at that point. Maybe git submodules will enable us to get the best of both worlds - I'll blog about it once we try ;)

The `.shared` folder

When dealing with multiple services, chances are high that some of those services will be configured similarly, e.g. for

installing common software
setting up unix users (with the same ids)
configuration (think php-cli for workers and php-fpm for web requests)

To avoid duplication, I place scripts (simple bash files) and config files in the .shared folder and make it available in the build context for each service. I'll explain the process in more detail under providing the correct build context.

`docker-test.sh`

Is really just a simple bash script that includes some high level tests to make sure that the containers are built correctly. See section Testing if everything works.

`.env.example` and `docker-compose.yml`

docker-compose uses a .env file for a convenient way to define and substitute environment variables. Since this .env file is environment specific, it is NOT
part of the repository (i.e. ignored via .gitignore). Instead, we provide a .env.example file that contains the required environment variables including reasonable default values. A new dev would usually run cp .env.example .env after checking out the repository for the first time. See section .env.example.

The Makefile `make` and `Makefile`s

are among those things that I've heard about occasionally but never really cared to understand (mostly because I associated them with C). Boy, did I miss out. I was comparing different strategies to provide code quality tooling (style checkers, static analyzers, tests, ...) and went from custom bash scripts over composer scripts to finally end up at Makefiles.

The Makefile serves as a central entry point and simplifies the management of the docker containers, e.g. for (re-)building, starting, stopping, logging in, etc. See section Makefile and .bashrc.

Defining services: php-fpm, nginx and workspace

Let's have a look at a real example and "refactor" the php-cli, php-fpm and nginx containers from the first part of this tutorial series.

This is the folder structure:

<project>/
├── .docker/
|   ├── .shared/
|   |   ├── config/
|   |   |   └── php/ 
|   |   |       └── conf.d/
|   |   |           └── zz-app.ini
|   |   └── scripts/
|   |       └── docker-entrypoint/
|   |           └── resolve-docker-host-ip.sh
|   ├── nginx/
|   |   ├── sites-available/
|   |   |   └── default.conf
|   |   ├── Dockerfile
|   |   └── nginx.conf
|   ├── php-fpm/
|   |   ├── php-fpm.d/
|   |   |   └── pool.conf
|   |   └── Dockerfile
|   ├── workspace/ (formerly php-cli)
|   |   ├── .ssh/
|   |   |   └── insecure_id_rsa
|   |   |   └── insecure_id_rsa.pub
|   |   └── Dockerfile
|   ├── .env.example
|   ├── docker-compose.yml
|   └── docker-test.sh
├── Makefile
└── index.php

php-fpm

Click here to see the full php-fpm Dockerfile.

Since we will be having two PHP containers, we need to place the common .ini settings in the .shared directory.

|   ├── .shared/
|   |   ├── config/
|   |   |   └── php/ 
|   |   |       └── conf.d/
|   |   |           └── zz-app.ini

For now, zz-app.ini will only contain our opcache setup:

; enable opcache
opcache.enable_cli = 1
opcache.enable = 1
opcache.fast_shutdown = 1
; revalidate everytime (effectively disabled for development)
opcache.validate_timestamps = 0

The pool configuration is only relevant for php-fpm, so it goes in the directory of the service. Btw. I highly recommend this video on PHP-FPM Configuration if your php-fpm foo isn't already over 9000.

|   ├── php-fpm/
|   |   ├── php-fpm.d/
|   |   |   └── pool.conf

Modifying the pool configuration

We're using the modify_config.sh script to set the user and group that owns the php-fpm processes.

# php-fpm pool config
COPY ${SERVICE_DIR}/php-fpm.d/* /usr/local/etc/php-fpm.d
RUN /tmp/scripts/modify_config.sh /usr/local/etc/php-fpm.d/zz-default.conf \
    "__APP_USER" \
    "${APP_USER}" \
 && /tmp/scripts/modify_config.sh /usr/local/etc/php-fpm.d/zz-default.conf \
    "__APP_GROUP" \
    "${APP_GROUP}" \
;

Custom ENTRYPOINT

Since php-fpm needs to be debuggable, we need to ensure that the host.docker.internal DNS entry exists, so we'll use the corresponding ENTRYPOINT to do that.

# entrypoint
RUN mkdir -p /bin/docker-entrypoint/ \
 && cp /tmp/scripts/docker-entrypoint/* /bin/docker-entrypoint/ \
 && chmod +x -R /bin/docker-entrypoint/ \
;

ENTRYPOINT ["/bin/docker-entrypoint/resolve-docker-host-ip.sh","php-fpm"]

nginx

Click here to see the full nginx Dockerfile.

The nginx setup is even simpler. There is no shared config, so that everything we need resides in

|   ├── nginx/
|   |   ├── sites-available/
|   |   |   └── default.conf
|   |   ├── Dockerfile
|   |   └── nginx.conf

Please note, that nginx only has the nginx.conf file for configuration (i.e. there is no conf.d directory or so), so we need to define the full config in there.

user __APP_USER __APP_GROUP;
worker_processes 4;
pid /run/nginx.pid;
daemon off;

http {
  # ...

  include /etc/nginx/sites-available/*.conf;

  # ...
}

There are two things to note:

user and group are modified dynamically
we specify /etc/nginx/sites-available/ as the directory that holds the config files for the individual files via include /etc/nginx/sites-available/*.conf; We need to keep the last point in mind, because we must use the same directory in the Dockerfile:

# nginx app config
COPY ${SERVICE_DIR}/sites-available/* /etc/nginx/sites-available/

The site's config file default.conf has a variable (__NGINX_ROOT) for the root directive and we "connect" it with the fpm-container via fastcgi_pass php-fpm:9000;

server {
    # ...
    root __NGINX_ROOT;
    # ...

    location ~ \.php$ {
        # ...
        fastcgi_pass php-fpm:9000;
    }
}

php-fpm will resolve to the php-fpm container, because we use php-fpm as the service name in the docker-compose file, so it will be automatically used as the hostname:

Other containers on the same network can use either the service name or [an] alias to connect to one of the service’s containers.

In the Dockerfile, we use

ARG APP_CODE_PATH
RUN /tmp/scripts/modify_config.sh /etc/nginx/sites-available/default.conf \
    "__NGINX_ROOT" \
    "${APP_CODE_PATH}" \
;

APP_CODE_PATH will be passed via docker-compose when we build the container and mounted as a shared directory from the host system.

workspace (formerly php-cli)

Click here to see the full workspace Dockerfile.

We will use the former php-cli container and make it our workspace as introduced in part 2 of this tutorial under Preparing the "workspace" container.

This will be the container we use to point our IDE to, e.g. to execute tests. Its Dockerfile looks almost identical to the one of the php-fpm service, apart from the SSH setup:

# set up ssh
RUN apt-get update -yqq && apt-get install -yqq openssh-server \
 && mkdir /var/run/sshd \
;

# add default public key to authorized_keys
USER ${APP_USER}
COPY ${SERVICE_DIR}/.ssh/insecure_id_rsa.pub /tmp/insecure_id_rsa.pub
RUN mkdir -p ~/.ssh \
 && cat /tmp/insecure_id_rsa.pub >> ~/.ssh/authorized_keys \
 && chown -R ${APP_USER}: ~/.ssh \
 && chmod 700 ~/.ssh \
 && chmod 600 ~/.ssh/authorized_keys \
;
USER root

Setting up docker-compose

In order to orchestrate the build process, we'll use docker-compose.

docker-compose.yml

See the full docker-compose.yml file in the repository

Things to note:

each service uses context: . so it has access to the .shared folder. The context is always relative to the location of the first docker-compose.yml file
all arguments that we used in the Dockerfiles are defined in the args: section via

  args:
    - APP_CODE_PATH=${APP_CODE_PATH_CONTAINER}
    - APP_GROUP=${APP_GROUP}
    - APP_GROUP_ID=${APP_GROUP_ID}
    - APP_USER=${APP_USER}
    - APP_USER_ID=${APP_USER_ID}
    - TZ=${TIMEZONE}

the codebase is synced from the host in all containers via

  volumes:
    - ${APP_CODE_PATH_HOST}:${APP_CODE_PATH_CONTAINER}

the nginx service exposes ports on the host machine so that we can access the containers from "outside" via

  ports:
    - "${NGINX_HOST_HTTP_PORT}:80"
    - "${NGINX_HOST_HTTPS_PORT}:443"

all services are part of the backend network so they can talk to each other. The nginx service has an additional alias that allows us to define an arbitrary host name via

  networks:
    backend:
      aliases:
        - ${APP_HOST}

I prefer to have a dedicated hostname per project (e.g. docker-php-tutorial.local)
instead of using 127.0.0.1 or localhost directly

.env.example

To fill in all the required variables / arguments, we're using a .env.example file with the following content:

# Default settings for docker-compose
COMPOSE_PROJECT_NAME=docker-php-tutorial
COMPOSE_FILE=docker-compose.yml
COMPOSE_CONVERT_WINDOWS_PATHS=1

# build
PHP_VERSION=7.3
TIMEZONE=UTC
NETWORKS_DRIVER=bridge

# application
APP_USER=www-data
APP_GROUP=www-data
APP_USER_ID=1000
APP_GROUP_ID=1000
APP_CODE_PATH_HOST=../
APP_CODE_PATH_CONTAINER=/var/www/current

# required so we can reach the nginx server from other containers via that hostname
APP_HOST=docker-php-tutorial.local

# nginx
NGINX_HOST_HTTP_PORT=80
NGINX_HOST_HTTPS_PORT=443

# workspace
WORKSPACE_HOST_SSH_PORT=2222

The COMPOSE_ variables in the beginning set some reasonable defaults for docker-compose.

Building and running the containers By now, we should have everything we need set up to get our dockerized PHP development up and running. If you haven't done it already, now would be a great time to clone the repository and checkout the `part_3_structuring-the-docker-setup-for-php-projects` branch:

git clone https://github.com/paslandau/docker-php-tutorial.git
cd docker-php-tutorial
git checkout part_3_structuring-the-docker-setup-for-php-projects

Now copy the .env.exmaple to .env. All the default values should work out of the box - unless you already have something running on port 80 or 443. In that case you have to change NGINX_HOST_HTTP_PORT / NGINX_HOST_HTTP_PORT to a free port.

cp .env.example .env

We can examine the "final" docker-compose.yml after the variable substitution via

docker-compose -f .docker/docker-compose.yml --project-directory .docker config

networks:
  backend:
    driver: bridge
services:
  nginx:
    build:
      args:
        APP_CODE_PATH: /var/www/current
        APP_GROUP: www-data
        APP_GROUP_ID: '1000'
        APP_USER: www-data
        APP_USER_ID: '1000'
        TZ: UTC
      context: D:\codebase\docker-php-tutorial\.docker
      dockerfile: ./nginx/Dockerfile
    image: php-docker-tutorial/nginx
    networks:
      backend:
        aliases:
        - docker-php-tutorial.local
    ports:
    - published: 80
      target: 80
    - published: 443
      target: 443
    volumes:
    - /d/codebase/docker-php-tutorial:/var/www/current:rw
  php-fpm:
// ...

Note, that this command is run from ./docker-php-tutorial. If we would run this from ./docker-php-tutorial/.docker, we could simply use docker-compose config - but since we'll define that in a Makefile later anyway, the additional "verbosity" won't matter ;)

This command is also a great way to check the various paths that are resolved to their absolute form, e.g.

context: D:\codebase\docker-php-tutorial\.docker

and

volumes:
- /d/codebase/docker-php-tutorial:/var/www/current:rw

The actual build is triggered via

docker-compose -f .docker/docker-compose.yml --project-directory .docker build --parallel

Since we have more than one container, it makes sense to build with --parallel.

To start the containers, we use

docker-compose -f .docker/docker-compose.yml --project-directory .docker up -d

and should see

$ docker-compose -f .docker/docker-compose.yml --project-directory .docker up -d
Starting docker-php-tutorial_nginx_1     ... done
Starting docker-php-tutorial_workspace_1 ... done
Starting docker-php-tutorial_php-fpm_1   ... done

Testing

if everything works After rewriting our own docker setup a couple of times, I've come to appreciate a structured way to test if "everything" works. Everything as in:

are all containers running?
does "host.docker.internal" exist?
do we see the correct output when sending a request to nginx/php-fpm?
are all required php extensions installed?

This might seem superfluous (after all, we just defined excatly that in the Dockerfiles), but there will come a time when you (or someone else) need to make changes (new PHP version, new extensions, etc.) and having something that runs automatically and informs you about obvious flaws is a real time saver.

You can see the full test file in the repository. Since my bash isn't the best, I try to keep it as simple as possible. The tests can be run via

sh .docker/docker-test.sh

and should yield something like this:

Testing service 'workspace'
=======
Checking if 'workspace' has a running container
OK
Testing PHP version '7.3' on 'workspace' for 'php' and expect to see 'PHP 7.3'
OK
Testing PHP module 'xdebug' on 'workspace' for 'php'
OK
Testing PHP module 'Zend OPcache' on 'workspace' for 'php'
OK
Checking 'host.docker.internal' on 'workspace'
OK

Testing service 'php-fpm'
=======
...

Makefile and `.bashrc`

In the previous sections I have introduced a couple of commands, e.g. for building and running containers. And to be honest, I find it kinda challenging to keep them in mind without having to look up the exact options and arguments. I would usually create a helper function or an alias in my local .bashrc file in a situation like that - but that wouldn't be available to other members of the team then and it would be very specific to this one project. Instead we'll provide a Makefile as a central reference point.

Using `make` as central entry point

Please refer to the repository for the full Makefile.

Going into the details of make is a little out of scope for this article, so I kindly refer to some articles that helped me get started:

Both are written with a PHP context in mind. Tip: If you are using PhpStorm, give the Makefile support plugin a try. And don't forget the number one rule: A Makefile requires tabs!

Note: If you are using Windows, make is probably not available. See Install make on Windows (MinGW) for instructions to set it up.

The Makefile ist located in the root of the application. Since we use a help target that makes the Makefile self-documenting, we can simply run make to see all the available commands:

$ make

Usage:
  make <target>

[Docker] Build / Infrastructure
  docker-clean                 Remove the .env file for docker
  docker-init                  Make sure the .env file exists for docker
  docker-build-from-scratch    Build all docker images from scratch, without cache etc. Build a specific image by providing the service name via: make docker-build CONTAINER=<service>
  docker-build                 Build all docker images. Build a specific image by providing the service name via: make docker-build CONTAINER=<service>
  docker-up                    Start all docker containers. To only start one container, use CONTAINER=<service>
  docker-down                  Stop all docker containers. To only stop one container, use CONTAINER=<service>
  docker-test                  Run the infrastructure tests for the docker setup

As a new developer, your "onboarding" to get a running infrastructure should now look like this:

make docker-clean
make docker-init
make docker-build-from-scratch
make docker-test

Install make on Windows (MinGW)

make doesn't exist on Windows and is also not part of the standard installation of MinGW (click here to learn how to setup MinGW) Setting is up is straight forward but as with "everything UI" it's easier if you can actually "see what I'm doing" - so here's a video:

The steps are as follows:

Set up mingw-get
- Instructions: https://web.archive.org/web/20200226035236/http://www.mingw.org/wiki/getting_started#toc5
- Download: https://sourceforge.net/projects/mingw/files/Installer/mingw-get-setup.exe/download
- Install and add the bin/ directory to PATH (shortcut systempropertiesadvanced). Notes:
  - Do not use an installation path that contains spaces!
  - The installation path can be different from your MinGW location
Install mingw32-make via

  mingw-get install mingw32-make

create the file bin/make with the content
```
mingw32-make.exe $*
```
Note: Sometimes Windows won't recognize non-.exe files - so instead of bin/make you might need to name the file
bin/make.exe (with the same content)
- Open a new shell and type make. The output should look something like this

  $ make
  mingw32-make: *** Keine Targets angegeben und keine ¦make¦-Steuerdatei gefunden.  Schluss.

Easy container access via `din` .bashrc helper

I've got one last goodie for working with Docker that I use all the time:

Logging into a running container via docker exec and the din / dshell helper.

To make this work, put the following code in your .bashrc file

function din() {
  filter=$1

  user=""
  if [[ -n "$2" ]];
  then
    user="--user $2"
  fi

  shell="bash"
  if [[ -n "$3" ]];
  then
    shell=$3
  fi

  prefix=""
  if [[ "$(expr substr $(uname -s) 1 5)" == "MINGW" ]]; then
    prefix="winpty"
  fi
  ${prefix} docker exec -it ${user} $(docker ps --filter name=${filter} -q | head -1) ${shell}
}

The $(docker ps --filter name=${filter} -q | head -1) part will find partial matches on running containers for the first argument and pass the result to the docker exec command. In effect, we can log into any container by only providing a minimal matching string on the container name. E.g. to log in the workspace container I can now simply type din works from anywhere on my system.

Fundamentals on building the containers

Since we have now "seen" the end result, let's take a closer look behind the scenes. I assume that you are already somewhat familiar with Dockerfiles and have used docker-compose to orchestrate multiple services (if not, check out Persisting image changes with a Dockerfile and Putting it all together: Meet docker-compose). But there are some points I would like to cover in a little more detail.

Understanding build context

There are two essential parts when building a container:

the Dockerfile
the build context

You can read about the official description in the Dockerfile reference. You'll usually see something like this:

docker build .

which assumes that you use the current directory as build context and that there is a Dockerfile in the same directory.

But you can also start the build via

docker build .docker -f .docker/nginx/Dockerfile
                 |      |
                 |      └── use the Dockerfile at ".docker/nginx/Dockerfile"
                 |
                 └── use the .docker subdirectory as build context

For me, the gist is this: The build context defines the files and folders (recursively) on your machine that are send from the Docker CLI to the Docker Daemon that executes the build process of a container so that you can reference those files in the Dockerfile (e.g. via COPY). Take the following structure for example:

<project>/
├── .docker/
    ├── .shared/
    |   └── scripts/
    |       └── ...
    └── nginx/
        ├── nginx.conf
        └── Dockerfile

Assume, that the current working directory is <project>/. If we started a build via

docker build .docker/nginx -f .docker/nginx/Dockerfile

the context would not include the .shared folder so we wouldn't be able to COPY the scripts/ subfolder. If we ran

docker build .docker -f .docker/nginx/Dockerfile

however, that would make the .shared folder available. In the Dockerfile itself, I need to know what the build context is, because I need to adjust the paths accordingly. Concrete example for the folder structure above and build triggered via docker build .docker -f .docker/nginx/Dockerfile:

FROM:nginx

# build context is .docker ...

# ... so the following COPY refers to .docker/.shared
COPY ./.shared /tmp

# ... so the following COPY refers to .docker/nginx/nginx.conf
COPY ./nginx/nginx.conf /tmp

The build context for all of our containers will be the .docker directory, so that all build processes have access to the .shared scripts and config. Yes, that also means that the php-fpm container has access to files that are only relevant to the mysql container (for instance), but the performance penalty is absolutely neglectable. Plus, as long as we don't actively COPY those irrelevant files, they won't bloat up our images.

A couple of notes:

I used to think that the build context is always tied to the location of the Dockerfile but that's only the default, it can be any directory
the build context is actually send to the build process - i.e. you should avoid unnecessary files / folders as this might affect performance, especially on big files (iaw: don't use / as context!)
similar to git, Docker knows the concept of a .dockerignore file to exclude files from being included in the build context

Dockerfile template

The Dockerfiles for the containers roughly follow the structure outlined below:

FROM ...

# path to the directory where the Dockerfile lives relative to the build context
ARG SERVICE_DIR="./service"

# get the scripts from the build context and make sure they are executable
COPY .shared/scripts/ /tmp/scripts/
RUN chmod +x -R /tmp/scripts/

# set timezone
ARG TZ=UTC
RUN /tmp/scripts/set_timezone.sh ${TZ}

# add users
ARG APP_USER=www-data
ARG APP_USER_ID=1000
ARG APP_GROUP=$(APP_USER)
ARG APP_GROUP_ID=$(APP_USER_ID)

RUN /tmp/scripts/create_user.sh ${APP_USER} ${APP_GROUP} ${APP_USER_ID} ${APP_GROUP_ID}

# install common software
RUN /tmp/scripts/install_software.sh

# perform any other, container specific build steps
COPY ${SERVICE_DIR}/config/* /etc/service/config
RUN /tmp/scripts/modify_config.sh /etc/service/config/default.conf \
    "__APP_USER" \
    "${APP_USER}" \
;
# [...]

# set default work directory
WORKDIR "..."

# cleanup 
RUN /tmp/scripts/cleanup.sh

# define ENTRYPOINT
ENTRYPOINT [...]
CMD [...]

The comments should suffice to give you an overview - so let's talk about the individual parts in detail.

Setting the timezone

Script: set_timezone.sh

Let's start with a simple and obvious one: Ensuring that all containers use the same system timezone (see here and here)

#!/bin/sh

TZ=$1
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

The script is then called from the Dockerfile via

ARG TZ=UTC
RUN /tmp/scripts/set_timezone.sh ${TZ}

Synchronizing file and folder ownership on shared volumes

Script: create_user.sh

Docker makes it really easy to share files between containers by using volumes. For simplicities sake, you can picture a volume simply as an additional disk that multiple containers have access to. And since it's PHP we're talking about here, sharing the same application files is a common requirement (e.g. for php-fpm, nginx, php-workers).

As long as you are only dealing with one container, life is easy: You can simply chown files to the correct user. But since the containers might have a different user setup, permissions/ownership becomes a problem. Checkout this video on Docker & File Permissions for a practical example in a Laravel application.

The first thing for me was understanding that file ownership does not depend on the user name but rather on the user id. And you might have guessed it: Two containers might have a user with the same name but with a different id. The same is true for groups, btw. You can check the id by running id <name>, e.g.

id www-data
uid=33(www-data) gid=33(www-data) groups=33(www-data)

That's inconvenient but rather easy to solve in most cases, because we have full control over the containers and can assign ids as we like (using usermod -u <id> <name>) and thus making sure every container uses the same user names with the same user ids.

Things get complicated when the volume isn't just a Docker volume but a shared folder on the host. This is usually what we want for development, so that changes on the host are immediately reflected in all the containers.

This issue only affects users with a linux host system! Docker Desktop (previously known as Docker for Mac / Docker for Win) has a virtualization layer in between that will effectively erase all ownership settings and make everything shared from the host available to every user in a container.

We use the following script to ensure a consistent user setup when building a container:

#!/bin/sh

APP_USER=$1
APP_GROUP=$2
APP_USER_ID=$3
APP_GROUP_ID=$4

new_user_id_exists=$(id ${APP_USER_ID} > /dev/null 2>&1; echo $?) 
if [ "$new_user_id_exists" = "0" ]; then
    (>&2 echo "ERROR: APP_USER_ID $APP_USER_ID already exists - Aborting!");
    exit 1;
fi

new_group_id_exists=$(getent group ${APP_GROUP_ID} > /dev/null 2>&1; echo $?) 
if [ "$new_group_id_exists" = "0" ]; then
    (>&2 echo "ERROR: APP_GROUP_ID $APP_GROUP_ID already exists - Aborting!");
    exit 1;
fi

old_user_id=$(id -u ${APP_USER})
old_user_exists=$(id -u ${APP_USER} > /dev/null 2>&1; echo $?) 
old_group_id=$(getent group ${APP_GROUP} | cut -d: -f3)
old_group_exists=$(getent group ${APP_GROUP} > /dev/null 2>&1; echo $?)

if [ "$old_group_id" != "${APP_GROUP_ID}" ]; then
    # create the group
    groupadd -f ${APP_GROUP}
    # and the correct id
    groupmod -g ${APP_GROUP_ID} ${APP_GROUP}
    if [ "$old_group_exists" = "0" ]; then
        # set the permissions of all "old" files and folder to the new group
        find / -group $old_group_id -exec chgrp -h ${APP_GROUP} {} \; || true
    fi 
fi

if [ "$old_user_id" != "${APP_USER_ID}" ]; then
    # create the user if it does not exist
    if [ "$old_user_exists" != "0" ]; then
        useradd ${APP_USER} -g ${APP_GROUP}
    fi

    # make sure the home directory exists with the correct permissions
    mkdir -p /home/${APP_USER} && chmod 755 /home/${APP_USER} && chown ${APP_USER}:${APP_GROUP} /home/${APP_USER} 

    # change the user id, set the home directory and make sure the user has a login shell
    usermod -u ${APP_USER_ID} -m -d /home/${APP_USER} ${APP_USER} -s $(which bash)

    if [ "$old_user_exists" = "0" ]; then
        # set the permissions of all "old" files and folder to the new user 
        find / -user $old_user_id -exec chown -h ${APP_USER} {} \; || true
    fi
fi

The script is then called from the Dockerfile via

ARG APP_USER=www-data
ARG APP_USER_ID=1000
ARG APP_GROUP=$(APP_USER)
ARG APP_GROUP_ID=$(APP_USER_ID)

RUN /tmp/scripts/create_user.sh ${APP_USER} ${APP_GROUP} ${APP_USER_ID} ${APP_GROUP_ID}

The default values can be overridden by passing in the corresponding build args.

Linux users should use the user id of the user on their host system - for Docker Desktop users the defaults are fine.

Modifying configuration files

For most services we probably need some custom configuration settings, like

setting php.ini values
changing the default user of a service
changing the location of logfiles

There are a couple of common approaches to modify application configuration in docker and we are currently trying to stick to two rules: 1. provide additional files that override defaults if possible 2. change non-static values with a simple search and replace via sed during the container build

Providing additional config files

Most services allow the specification of additional configuration files that override the default values in a default config file. This is great because we only need to define the settings that we actually care about instead of copying a full file with lots of redundant values.

Take the php.ini file for example: It allows to places additional .ini files in a specific directory that override the default values. An easy way to find this directory is php -i | grep "additional .ini":

$ php -i | grep "additional .ini"
Scan this dir for additional .ini files => /usr/local/etc/php/conf.d

So instead of providing a "full" php.ini file, we will use a zz-app.ini file instead, that only contains the .ini settings we actually want to change and place it under /usr/local/etc/php/conf.d.

Why zz-? Because

[...] Within each directory, PHP will scan all files ending in .ini in alphabetical order.

so if we want to ensure that our .ini files comes last (overriding all previous settings), we'll give it a corresponding prefix :)

The full process would look like this:

place the file in the .docker folder, e.g. at .docker/.shared/config/php/conf.d/zz-app.ini
pass the folder as build context
in the Dockerfile, use COPY .shared/config/php/conf.d/zz-app.ini /usr/local/etc/php/conf.d/zz-app.ini

Changing non-static values

Script: modify_config.sh

Some configuration values are subject to local settings and thus should not be hard coded in configuration files. Take the memory_limit configuration for php-fpm as an example: Maybe someone in the team can only dedicate a limited amount of memory to docker, so the memory_limit has to be kept lower than usual.

We'll account for that fact by using a variable prefixed by __ instead of the real value and replace it with a dynamic argument in the Dockerfile. Example for the aforementioned zz-app.ini:

memory_limit = __MEMORY_LIMIT

We use the following script modify_config.sh to replace the value:

#!/bin/sh

CONFIG_FILE=$1
VAR_NAME=$2
VAR_VALUE=$3

sed -i -e "s#${VAR_NAME}#${VAR_VALUE}#" "${CONFIG_FILE}"

The script is then called from the Dockerfile via

ARG PHP_FPM_MEMORY_LIMIT=1024M

RUN /tmp/scripts/modify_config.sh \
    "/usr/local/etc/php/conf.d/zz-app.ini" \
    "__MEMORY_LIMIT" \
    "${PHP_FPM_MEMORY_LIMIT}" \
;

where PHP_FPM_MEMORY_LIMIT has a default value of 1024M but can be overridden when the actual build is initiated.

Installing php extensions

Script: install_php_extensions.sh

When php extensions are missing, googling will often point to answers for normal linux systems using apt-get or yum, e.g. sudo apt-get install php-xdebug. But for the official docker images, the recommended way is using the docker-php-ext-configure, docker-php-ext-install, and docker-php-ext-enable helper scripts. Unfortunately, some extensions have rather complicated dependencies, so that the installation fails. Fortunately, there is a great project on Github called docker-php-extension-installer that takes care of that for us and is super easy to use:

FROM php:7.3-cli

ADD https://raw.githubusercontent.com/mlocati/docker-php-extension-installer/master/install-php-extensions /usr/local/bin/

RUN chmod uga+x /usr/local/bin/install-php-extensions && sync && install-php-extensions xdebug

The readme also contains an overview of supported extension per PHP version. To ensure that all of our PHP containers have the same extensions, we provide the following script:

#!/bin/sh

# add wget
apt-get update -yqq && apt-get -f install -yyq wget

# download helper script
wget -q -O /usr/local/bin/install-php-extensions https://raw.githubusercontent.com/mlocati/docker-php-extension-installer/master/install-php-extensions \
    || (echo "Failed while downloading php extension installer!"; exit 1)

# install all required extensions
chmod uga+x /usr/local/bin/install-php-extensions && sync && install-php-extensions \
    xdebug \
    opcache \
;

If you're not sure which extensions are required by your application, give the ComposerRequireChecker a try.

Installing common software

Script: install_software.sh

There is a certain set of software that I want to have readily available in every container. Since this a development setup, I'd prioritize ease of use / debug over performance / image size, so this might seem like a little "too much". I think I'm also kinda spoiled by my Homestead past, because it's so damn convenient to have everything right at your fingertips :)

Anyway, the script is straight forward:

#!/bin/sh

apt-get update -yqq && apt-get install -yqq \
    curl \
    dnsutils \
    gdb \
    git \
    htop \
    iputils-ping \
    iproute2 \
    ltrace \
    make \
    procps \
    strace \
    sudo \
    sysstat \
    unzip \
    vim \
    wget \
;

Notes:

this list should match your own set of go-to tools. I'm fairly open to adding new stuff here if it speeds up the dev workflow. But if you don't require some of the tools, get rid of them.
sorting the software alphabetically is a good practice to avoid unnecessary duplicates. Don't do this by hand, though! If you're using an IDE / established text editor, chances are high that this is either a build-in functionality or there's a plugin available. I'm using Lines Sorter for PhpStorm

Cleaning up

Script: cleanup.sh

Nice and simple:

#!/bin/sh

apt-get clean
rm -rf /var/lib/apt/lists/* \
       /tmp/* \
       /var/tmp/* \
       /var/log/lastlog \
       /var/log/faillog

Using `ENTRYPOINT` for pre-run configuration

Docker went back to the unix roots with the do on thing and do it well philosophy which is manifested in the CMD and ENTRYPOINT instructions.

As I had a hard time understanding those instructions when I started with Docker, here's my take at a layman's terms description:

since a container should do one thing, we need to specify that thing. That's what we do with ENTRYPOINT. Concrete examples:
- a mysql container should probably run the mysqld daemon
- a php-fpm container.. well, php-fpm
the CMD is passed as the default argument to the ENTRYPOINT
the ENTRYPOINT is executed every time we run a container. Some things can't be done during build but only at runtime (e.g. find the IP of the host from within a container - see section Providing host.docker.internal for linux host systems ) - ENTRYPOINT is a good solution for that problem
technically, we can only override an already existing ENTRYPOINT from the base image. But: We can structure the new ENTRYPOINT like a decorator by adding exec "$@" at the end to simulate inheritance from the parent image

To expand on the last point, consider the default ENTRYPOINT of the current [2019-02-23; PHP 7.3] php-fpm image

#!/bin/sh

set -e

# first arg is `-f` or `--some-option`
if [ "${1#-}" != "$1" ]; then
    set -- php-fpm "$@"
fi

exec "$@"

In the corresponding Dockerfile we find the following instructions:

# [...]
ENTRYPOINT ["docker-php-entrypoint"]
# [...]
CMD ["php-fpm"]

That means: When we run the container it will pass the string "php-fpm" to the ENTRYPOINT script docker-php-entrypoint as argument which will then execute it (due to the exec "$@" instruction at the end):

$ docker run --name test --rm php:fpm
[23-Feb-2019 14:49:20] NOTICE: fpm is running, pid 1
[23-Feb-2019 14:49:20] NOTICE: ready to handle connections
# php-fpm is running
# Hit ctrl + c to close the connection
$ docker stop test

We could now override the default CMD "php-fpm" with something else, e.g. a simple echo "hello". The ENTRYPOINT will happily execute it:

$ docker run --name test --rm php:fpm echo "hello"
hello

But now the php-fpm process isn't started any more. How can we echo "hello" but still keep the fpm process running?
By adding our own ENTRYPOINT script:

#!/bin/sh
echo 'hello'

exec "$@"

Full example (using stdin to pass the Dockerfile via Heredoc string)

$ docker build -t my-fpm -<<'EOF'
FROM php:fpm

RUN  touch "/usr/bin/my-entrypoint.sh" \
  && echo "#!/bin/sh" >> "/usr/bin/my-entrypoint.sh" \
  && echo "echo 'hello'" >> "/usr/bin/my-entrypoint.sh" \
  && echo "exec \"\$@\"" >> "/usr/bin/my-entrypoint.sh" \
  && chmod +x "/usr/bin/my-entrypoint.sh" \
  && cat "/usr/bin/my-entrypoint.sh" \
;

ENTRYPOINT ["/usr/bin/my-entrypoint.sh", "docker-php-entrypoint"]
CMD ["php-fpm"]
EOF

Note that we added the ENTRYPOINT of the parent image docker-php-entrypoint as argument to our own ENTRYPOINT script /usr/bin/my-entrypoint.sh so that we don't loose its functionality. And we need to define the CMD instruction explicitly, because the one from the parent image is automatically removed once we define our own ENTRYPOINT.

But: It works:

$ docker run --name test --rm my-fpm
hello
[23-Feb-2019 15:43:25] NOTICE: fpm is running, pid 1
[23-Feb-2019 15:43:25] NOTICE: ready to handle connections
# Hit ctrl + c to close the connection
$ docker stop test

Providing `host.docker.internal` for linux host systems

Script: docker-entrypoint/resolve-docker-host-ip.sh

In the previous part of this tutorial series, I explained how to build the Docker container in a way that it plays nice with PhpStorm and Xdebug. The key parts were SSH access and the magical host.docker.internal DNS entry. This works great for Docker Desktop (Windows and Mac) but not for Linux. The DNS entry doesn't exist there. Since we rely on that entry to make debugging possible, we will set it "manually" if the host doesn't exist with the following script (inspired by the article Access host from a docker container):

#!/bin/sh

set -e

HOST_DOMAIN="host.docker.internal"

# check if the host exists - this will fail on linux
if dig ${HOST_DOMAIN} | grep -q 'NXDOMAIN'
then
  # resolve the host IP
  HOST_IP=$(ip route | awk 'NR==1 {print $3}')
  # and write it to the hosts file
  echo "$HOST_IP\t$HOST_DOMAIN" >> /etc/hosts
fi

exec "$@"

The script is placed at .shared/docker-entrypoint/resolve-docker-host-ip.sh and added as ENTRYPOINT in the Dockerfile via

COPY .shared/scripts/ /tmp/scripts/

RUN mkdir -p /bin/docker-entrypoint/ \
 && cp /tmp/scripts/docker-entrypoint/* /bin/docker-entrypoint/ \
 && chmod +x -R /bin/docker-entrypoint/ \
;

ENTRYPOINT ["/bin/docker-entrypoint/resolve-docker-host-ip.sh", ...]

Notes:

since this script depends on runtime configuration, we need to run it as an ENTRYPOINT
there is no need to explicitly check for the OS type - we simply make sure that the DNS entry exists and add it if it doesn't
we're using dig (package dnsutils) and ip (package iproute2) which need to be installed during the build time of the container. Tip: If you need to figure out the package for a specific command, give https://command-not-found.com/ a try. See the entry for dig for instance.
this workaround is only required in containers we want to debug via xdebug

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. Apart from that, you should now have a running docker setup for your local PHP development as well as a nice "flow" to get started each day.

In the next part of this tutorial, we will add some more containers (php workers, mysql, redis) and use a fresh installation of Laravel 9 to make use of them.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

PhpStorm, Docker and Xdebug 3 on PHP 8.1 in 2022 [Tutorial Part 3]

Pascal Landau — Fri, 24 Jun 2022 08:22:02 +0000

This article appeared first on https://www.pascallandau.com/ at PhpStorm, Docker and Xdebug 3 on PHP 8.1 in 2022 [Tutorial Part 3]

In this part of the tutorial series on developing PHP on Docker we will setup our local development environment to be used by PhpStorm and Xdebug. We will also ensure that we can run PHPUnit tests from the command line as well as from PhpStorm and throw the tool strace into the mix for debugging long running processes.

All code samples are publicly available in my Docker PHP Tutorial repository on Github. You find the branch for this tutorial at part-4-2-phpstorm-docker-xdebug-3-php-8-1-in-2022

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Docker from scratch for PHP 8.1 Applications in 2022 and the following one is Run Laravel 9 on Docker in 2022.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
Install Tools
- Install composer
- Install Xdebug
- Install PHPUnit
- Install SSH
Setup PhpStorm
- SSH Configuration
- PHP Interpreter
- PHPUnit
- Debugging
  - Debug code executed via PhpStorm
  - Debug code executed via php-fpm, cli or from a worker
    - php-fpm
    - cli
    - php-workers
strace
Wrapping up

Introduction

This article is mostly an update of Setting up PhpStorm with Xdebug for local development on Docker but will also cover the "remaining cases" of debugging php-fpm and php worker processes.

We will still rely on an always-running docker setup that we connect to via an SSH Configuration instead of using the built-in docker-compose capabilities as I feel it's closer to what we do in CI / production. However, we will not use SSH keys any longer but simply authenticate via password. This reduces complexity and removes any pesky warnings regarding "SSH keys being exposed in a repository".

Install Tools

Install composer Composer is installed by pulling the official composer docker image and simply "copying" the composer executable over to the base php image. In addition, composer needs the extensions `mbstring` and `phar`

# File: .docker/images/php/base/Dockerfile

ARG ALPINE_VERSION
ARG COMPOSER_VERSION
FROM composer:${COMPOSER_VERSION} as composer
FROM alpine:${ALPINE_VERSION} as base

# ...

RUN apk add --update --no-cache  \
        php-mbstring~=${TARGET_PHP_VERSION} \
        php-phar~=${TARGET_PHP_VERSION} \

# ...

COPY --from=composer /usr/bin/composer /usr/local/bin/composer

Because we want our build to be deterministic, we "pin" the composer version by adding a COMPOSER_VERSION variable to the .docker/.env file

COMPOSER_VERSION=2.2.5

and using it in .docker/docker-compose/docker-compose-php-base.yml:

services:
  php-base:
    build:
      args:
        - COMPOSER_VERSION=${COMPOSER_VERSION?}

Install Xdebug Install the extension via `apk` (only for the `local` target):

# File: .docker/images/php/base/Dockerfile

FROM base as local

RUN apk add --no-cache --update \
        php-xdebug~=${TARGET_PHP_VERSION} \
    # ensure that xdebug is not enabled by default
    && rm -f /etc/php8/conf.d/00_xdebug.ini

We also don't want to enable xdebug immediately but only when we need it (due to the decrease in performance when the extension is enabled), hence we remove the default config file and disable the extension in the application .ini file

# File: .docker/images/php/base/conf.d/zz-app-local.ini

; Note:
; Remove the comment ; to enable debugging
;zend_extension=xdebug
xdebug.client_host=host.docker.internal
xdebug.start_with_request=yes
xdebug.mode=debug

See Fix Xdebug on PhpStorm when run from a Docker container for an explanation of the xdebug.client_host=host.docker.internal setting (previously called xdebug.remote_host in xdebug < 3). This will still work out of the box for Docker Desktop, but for Linux users we need to add the host-gateway magic reference
to all PHP containers (we can't add it to the php base image because this is a runtime setting):

services:
  service:
    extra_hosts:
      - host.docker.internal:host-gateway

Finally, we need to add the environment variable PHP_IDE_CONFIG
to all PHP containers. The variable is defined as PHP_IDE_CONFIG=serverName=dofroscra, where "dofroscra" is the name of the server that we will configure later for debugging. Because we need the same value in multiple places, the variable is configured in .docker/.env:

PHP_IDE_CONFIG=serverName=dofroscra

And then added in .docker/docker-compose/docker-compose.local.yml

services:
  php-fpm:
    environment:
      - PHP_IDE_CONFIG=${PHP_IDE_CONFIG?}

  php-worker:
    environment:
      - PHP_IDE_CONFIG=${PHP_IDE_CONFIG?}

  application:
    environment:
      - PHP_IDE_CONFIG=${PHP_IDE_CONFIG?}

Install PHPUnit PHPUnit will be installed via `composer` but will not be "baked into the image" for local development. Thus, we must run `composer require` in the container. To make this more convenient a make target for running arbitrary composer commands is added in `.make/01-00-application-setup.mk`:

.PHONY: composer
composer: ## Run composer commands. Specify the command e.g. via ARGS="install"
    $(EXECUTE_IN_APPLICATION_CONTAINER) composer $(ARGS);

This allows me to run make composer ARGS="install" from the host system to execute composer install in the container. In consequence, composer will use the PHP version and extensions of the application container to install the dependencies, yet I will still see the installed files locally because the codebase is configured as a volume for the container.

Before installing phpunit, we must add the required extensions dom and xml to the container

# File: .docker/images/php/base/Dockerfile

# ...

RUN apk add --update --no-cache  \
        php-dom~=${TARGET_PHP_VERSION} \
        php-xml~=${TARGET_PHP_VERSION} \

as well as rebuild and restart the docker setup via

make docker-build
make docker-down
make docker-up

Now we can add phpunit via

make composer ARGS='require "phpunit/phpunit"'

which will create a composer.json file and setup up the vendor/ directory:

$ make composer ARGS='require "phpunit/phpunit"'
Using version ^9.5 for phpunit/phpunit
./composer.json has been created
Running composer update phpunit/phpunit
Loading composer repositories with package information
Updating dependencies
...

CAUTION: If you run into the following permission error at this step, you are likely using Linux and haven't set the APP_USER_ID and APP_GROUP_ID variables as described in the previous article under Solving permission issues.

make composer ARGS='req phpunit/phpunit' ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application application composer req phpunit/phpunit
./composer.json is not writable.
make: *** [.make/01-00-application-setup.mk:14: composer] Error 1

I have also added

a minimal phpunit.xml config file
a test case at tests/SomeTest.php
and a new Makefile for "anything related to qa" at .make/01-02-application-qa.mk:

##@ [Application: QA]

.PHONY: test
test: ## Run the test suite 
    $(EXECUTE_IN_WORKER_CONTAINER) vendor/bin/phpunit -c phpunit.xml

So I can run tests simply via make test

$ make test
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker vendor/bin/phpunit
PHPUnit 9.5.13 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 00:00.324, Memory: 4.00 MB

OK (1 test, 1 assertion)

Install SSH

We will execute commands from PhpStorm via ssh in the application container. As mentioned, we won't use a key file for authentication but will instead simply use a password that is configured via the APP_SSH_PASSWORD variable in .docker/.env and passed to the image in .docker/docker-compose/docker-compose.local.yml. In addition, we map port 2222 from the host system to port 22 of the application container and make sure that the codebase is shared as a volume between host and container

  application:
    build:
      args:
        - APP_SSH_PASSWORD=${APP_SSH_PASSWORD?}
    volumes:
      - ${APP_CODE_PATH_HOST?}:${APP_CODE_PATH_CONTAINER?}
    ports:
      - "${APPLICATION_SSH_HOST_PORT:-2222}:22"

The container already contains openssh and sets the password

ARG BASE_IMAGE
FROM ${BASE_IMAGE} as base

FROM base as local

RUN apk add --no-cache --update \
        openssh

ARG APP_SSH_PASSWORD
RUN echo "$APP_USER_NAME:$APP_SSH_PASSWORD" | chpasswd 2>&1

# Required to start sshd, otherwise the container will error out on startup with the message
# "sshd: no hostkeys available -- exiting."
# @see https://stackoverflow.com/a/65348102/413531 
RUN ssh-keygen -A

# we use SSH deployment configuration in PhpStorm for local development
EXPOSE 22

CMD ["/usr/sbin/sshd", "-D"]

Setup PhpStorm

We will configure a remote PHP interpreter that uses an SSH connection to run commands in the application container. Before, we have been using an SFTP Deployment configuration , which was kinda confusing ("What is SFTP doing here?"), so we will use an SSH Configuration instead and configure the path mappings in the Cli Interpreter interface

SSH Configuration

At File | Settings | Tools | SSH Configurations create a new SSH Configuration named "Docker PHP Tutorial" with the following settings

Host: 127.0.0.1
Port: see APPLICATION_SSH_HOST_PORT in .docker/docker-compose/docker-compose.local.yml
User name: see APP_USER_NAME in .make/.env
Authentication type: Password
Password: see APP_SSH_PASSWORD in .docker/.env

PHP Interpreter

At File | Settings | PHP add a new PHP CLI interpreter that uses the new SSH Configuration

In addition, we define the path to the xdebug extension because it is disabled by default but PhpStorm can enable it automatically if required. You can find the path in the application container via

root:/var/www/app# php -i | grep extension_dir
extension_dir => /usr/lib/php8/modules => /usr/lib/php8/modules
root:/var/www/app# ll /usr/lib/php8/modules | grep xdebug
-rwxr-xr-x    1 root     root        303936 Jan  9 00:21 xdebug.so

We still need to Fix Xdebug on PhpStorm when run from a Docker container by adding a custom PHP option for xdebug.client_host=host.docker.internal. That's the same value we use in .docker/images/php/base/conf.d/zz-app-local.ini.

In the interpreter overview we must now configure the path mappings so that PhpStorm knows "which local file belongs to which remote one". The remote folder is defined in .docker/.env via

APP_CODE_PATH_CONTAINER=/var/www/app

Afterwards we can set a breakpoint e.g. in setup.php and start debugging:

The screenshot shows that PhpStorm adds the Xdebug extension that we defined previously.

PHPUnit

phpunit is configured via File | Settings | PHP | Test Frameworks. First, we select the interpreter that we just added

Then, we add the paths to the composer autoload script and the phpunit.xml configuration file.

PhpStorm will now execute tests using the PHP interpreter in the application container

Debugging

First of all, if you haven't already please also take a look at the official xdebug documentation. Derick is doing a great job at explaining xdebug in detail including some helpful videos like Xdebug 3: Xdebug with Docker and PhpStorm in 5 minutes

Debug code executed via PhpStorm

This should already work out of the box. Simply set a break point, right-click on a file and choose "Debug '...'"

Debug code executed via php-fpm, cli or from a worker

For code that is executed "directly" by a container without PhpStorm, we first need to enable xdebug in the container by removing the ; in front of the extension in /etc/php8/conf.d/zz-app-local.ini

; Note:
; Remove the comment ; to enable debugging
zend_extension=xdebug

To make this a little more convenient, we use dedicated make recipes for those actions in .make/01-01-application-commands.mk

.PHONY: execute-in-container
execute-in-container: ## Execute a command in a container. E.g. via "make execute-in-container DOCKER_SERVICE_NAME=php-fpm COMMAND="echo 'hello'"
    @$(if $(DOCKER_SERVICE_NAME),,$(error DOCKER_SERVICE_NAME is undefined))
    @$(if $(COMMAND),,$(error COMMAND is undefined))
    $(EXECUTE_IN_CONTAINER) $(COMMAND);

.PHONY: enable-xdebug
enable-xdebug: ## Enable xdebug in the given container specified by "DOCKER_SERVICE_NAME". E.g. "make enable-xdebug DOCKER_SERVICE_NAME=php-fpm"
    "$(MAKE)" execute-in-container APP_USER_NAME="root" DOCKER_SERVICE_NAME=$(DOCKER_SERVICE_NAME) COMMAND="sed -i 's/.*zend_extension=xdebug/zend_extension=xdebug/' '/etc/php8/conf.d/zz-app-local.ini'"

.PHONY: disable-xdebug
disable-xdebug: ## Disable xdebug in the given container specified by "DOCKER_SERVICE_NAME". E.g. "make enable-xdebug DOCKER_SERVICE_NAME=php-fpm"
    "$(MAKE)" execute-in-container APP_USER_NAME="root" DOCKER_SERVICE_NAME=$(DOCKER_SERVICE_NAME) COMMAND="sed -i 's/.*zend_extension=xdebug/;zend_extension=xdebug/' '/etc/php8/conf.d/zz-app-local.ini'"

To capture incoming requests, we need to make PhpStorm listen for PHP Debug connections via Run | Start Listening for PHP Debug Connections.

The corresponding ports are configured at File | Settings | PHP | Debug. In Xdebug < 3 the default port was 9000 and in Xdebug 3 it is 9003

Finally, we need to add a server via File | Settings | PHP | Servers

The name of the server must match the value of the serverName key in the environment variable PHP_IDE_CONFIG that we configured previously as serverName=dofroscra.

php-fpm

For php-fpm we must restart the php-fpm process without restarting the container after we have activated xdebug via

kill -USR2 1

Since this is a pain to remember, we add a make target in .make/01-01-application-commands.mk

# @see https://stackoverflow.com/a/43076457
.PHONY: restart-php-fpm
restart-php-fpm: ## Restart the php-fpm service
    "$(MAKE)" execute-in-container DOCKER_SERVICE_NAME=$(DOCKER_SERVICE_NAME_PHP_FPM) COMMAND="kill -USR2 1"

So we can now simply run

make enable-xdebug DOCKER_SERVICE_NAME=php-fpm
make restart-php-fpm

Setting a breakpoint in public/index.php and opening http://127.0.0.1/ in a browser or via curl http://127.0.0.1/ will halt the execution as expected.

cli

Instead of triggering a PHP script via HTTP request, we can also run CLI scripts - think of the make setup-db target for instance. To debug such invocations, we need to follow the same steps as before:

enable the xdebug extension in the application container
"Listening for PHP Debug Connections" from PhpStorm

Running the following make targets will trigger a breakpoint in setup.php:

make enable-xdebug DOCKER_SERVICE_NAME=application
make setup-db

php-workers

And finally the same thing for long running PHP processes (aka workers). Just as before:

enable the xdebug extension in the php-worker container
"Listening for PHP Debug Connections" from PhpStorm
restart the php workers

Running the following make targets will trigger a breakpoint in worker.php:

make enable-xdebug DOCKER_SERVICE_NAME=php-worker
make restart-workers

strace

strace is a great tool for debugging long running processes that I've adopted after reading What is PHP doing?. I've added it to the php base image:

RUN apk add --update --no-cache \
        strace

You can attach to any running process via sudo strace -p $processId - BUT that doesn't work out of the box on docker and will fail with the error message

strace: attach: ptrace(PTRACE_SEIZE, 1): Operation not permitted

This is caused by a security measure from docker and can be circumvented by adding

services:
  service:
    cap_add:
      - "SYS_PTRACE"
    security_opt:
      - "seccomp=unconfined"

in .docker/docker-compose/docker-compose.local.yml to all PHP containers. After rebuilding and restarting the docker setup, you can now e.g. log in the php-worker container and run strace on a php worker process:

application:/var/www/app# ps aux
PID   USER     TIME  COMMAND
    1 applicat  0:00 {supervisord} /usr/bin/python3 /usr/bin/supervisord
    7 applicat  0:00 php /var/www/app/worker.php
    8 applicat  0:00 php /var/www/app/worker.php
    9 applicat  0:00 php /var/www/app/worker.php
   10 applicat  0:00 php /var/www/app/worker.php
   11 applicat  0:00 bash
   20 applicat  0:00 ps aux
application:/var/www/app# sudo strace -p 7
strace: Process 7 attached
restart_syscall(<... resuming interrupted read ...>) = 0
poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(4, "*2\r\n$4\r\nRPOP\r\n$5\r\nqueue\r\n", 25, MSG_DONTWAIT, NULL, 0) = 25
poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 1 ([{fd=4, revents=POLLIN}])
recvfrom(4, "$", 1, MSG_PEEK, NULL, NULL) = 1

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. Apart from that, you should now have a fully configured development setup that works with PhpStorm as your IDE.

In the next part of this tutorial, we will use a fresh installation of Laravel on top of our setup.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

Docker from scratch for PHP 8.1 Applications in 2022 [Tutorial Part 2]

Pascal Landau — Mon, 20 Jun 2022 07:40:19 +0000

This article appeared first on https://www.pascallandau.com/ at Docker from scratch for PHP 8.1 Applications in 2022 [Tutorial Part 2]

In this part of the tutorial series on developing PHP on Docker we will revisit the previous tutorials and update some things to be up-to-date in 2022.

All code samples are publicly available in my Docker PHP Tutorial repository on Github. You find the branch for this tutorial at part-4-1-docker-from-scratch-for-php-applications-in-2022

All published parts of the Docker PHP Tutorial are collected under a dedicated page at Docker PHP Tutorial. The previous part was Structuring the Docker setup for PHP Projects and the following one is PhpStorm, Docker and Xdebug 3 on PHP 8.1 in 2022.

If you want to follow along, please subscribe to the RSS feed or via email to get automatic notifications when the next part comes out :)

Introduction
Local docker setup
Docker
- docker-compose
  - .docker/.env file and required ENV variables
- Images
  - PHP images
    - ENV vs ARG
  - Image naming convention
  - Environments and build targets
Makefile
- .make/*.mk includes
- Shared variables: .make/.env
  - Manual modifications
- Enforce required parameters
Make + Docker = <3
- Ensuring the build order
- Run commands in the docker containers
  - Solving permission issues
PHP POC
Wrapping up

Introduction

If you have read the previous tutorial Structuring the Docker setup for PHP Projects you might encounter some significant changes. The tutorial was published over 2 years ago, Docker has evolved and I have learned more about it. Plus, I gathered practical experience (good and bad) with the previous setup. I would now consider most of the points under Fundamentals on building the containers as either "not required" or simply "overengineered / too complex". To be concrete:

Setting the timezone
- not required if the default is already UTC (which is almost always the case)
Synchronizing file and folder ownership on shared volumes
- this is only an issue if files need to be modified by containers and the host system - which is only really relevant for the PHP containers
- in addition, I would recommend adding a completely new user (e.g. application) instead of re-using an existing one like www-data - this simplifies the whole user setup a lot
- from now on we will be using application as the user name (APP_USER_NAME) and 10000 the user id (APP_USER_ID; following the best practice to not use a UID below 10,000)
Modifying configuration files
- just use sed - no need for a dedicated script
Installing php extensions
- see PHP images - will now be done via apk add
Installing common software
- see PHP images - since there is only one base image there is no need for a dedicated script
Cleaning up
- didn't really make sense because the "cleaned up files" were already part of a previous layer
- we might "bring it back" later when we optimize the image size to speed up the pushing/pulling of the images to/from the registry
Providing host.docker.internal for linux host systems
- can now be done via the host-gateway magic reference
```
services:
     myservice:
          extra_hosts:
            - host.docker.internal:host-gateway
```
- thus, no custom entrypoint is required any longer

Local docker setup

The goal of this part is the introduction of a working local setup without development tools. In other words: We want the bare minimum to have something running locally.

The main components are:

the make setup in the Makefile and in the .make/ directory
the docker setup in the .docker/ directory
some PHP files that act as a POC for the end2end functionality of the docker setup

Check out the code via

git checkout part_4_section_1_docker_from_scratch_for_php_applications_in_2022_code_structure

initialize it via

make make-init
make docker-build

and run it via

make docker-up

Now you can access the web interface via http://127.0.0.1. The following diagram shows how the containers are connected

See also the PHP POC for a full test of the setup.

Docker

The docker setup consists of

an nginx container as a webserver
a MySQL database container
a Redis container that acts as a queue
a php base image that is used by
- a php worker container that spawns multiple PHP worker processes via supervisor
- a php-fpm container as a backend for the nginx container
- an application container that we use to run commands

We keep the .docker/ directory from the previous tutorial , though it will be split into docker-compose/ and images/ like so:

.
└── .docker/
    ├── docker-compose/
    |   ├── docker-compose.yml
    |   └── <other docker-compose files>
    ├── images/
    |   ├── nginx/
    |   |   ├── Dockerfile
    |   |   └── <other files for the nginx image>
    |   └── <other folders for docker images>
    ├── .env
    └── .env.example

docker-compose

All images are build via docker-compose because the docker-compose.yml file(s) provide a nice abstraction layer for the build configuration. In addition, we can also use it to orchestrate the containers, i.e. control volumes, port mappings, networking, etc. - as well as start and stop them via docker-compose up and docker-compose down.

FYI: Even though it is convenient to use docker-compose for both things, I found it also to make the setup more complex than it needs to be when running things later in production (when we are not using docker-compose any longer). I believe the problem here is that some modifications are ONLY required for building while others are ONLY required for running - and combining both in the same file yields a certain amount of noise. But: It is what it is.

We use three separate docker-compose.yml files:

docker-compose.yml
- contains all information valid for all environments
docker-compose.local.yml
- contains information specific to the local environment, see Environments and build targets
docker-compose-php-base.yml
- contains information for building the php base image, see PHP images

`.docker/.env` file and required ENV variables

In our docker setup we basically have 3 different types of variables:

variables that depend on the local setup of an individual developer, e.g. the NGINX_HOST_HTTP_PORT on the host machine (because the default one might already be in use) 2. variables that are used in multiple images, e.g. the location of the codebase within a container's file system 3. variables that hold information that is "likely to change", e.g. the exact version of a base image

Since - again - we strive to retain a single source of truth, we extract the information as variables and put them in a .docker/.env file. In a perfect world, I would like to separate these different types in different files - but docker-compose only allows a single .env file, see e.g. this comment. If the file does not exist, it is copied from .docker/.env.example.

The variables are then used in the docker-compose.yml file(s). I found it to be "the least painful" to always use the ? modifier on variables so that docker-compose fails immediately if the variable is missing.

Note: Some variables are expected to be passed via environment variables when docker-compose is invoked (i.e. they are required but not defined in the .env file; see also Shared variables: .make/.env

Images

For MySQL and redis we do not use custom-built images but instead use the official ones directly and configure them through environment variables when starting the containers. In production, we won't use docker anyway for these services but instead rely on the managed versions, e.g.

redis => Memorystore for Redis (GCP) or ElastiCache für Redis (AWS)
mysql => Cloud SQL for MySQL (GCP) or RDS for MySQL (AWS)

The remaining containers are defined in their respective subdirectories in the .docker/images/ directory, e.g. the image for the nginx container is build via the Dockerfile located in .docker/images/nginx/Dockerfile.

PHP images

We need 3 different PHP images (fpm, workers, application) and use a slightly different approach than in Structuring the Docker setup for PHP Projects:

Instead of using the official PHP base images (i.e. cli or fpm), we use a "plain" alpine base image and install PHP and the required extensions manually in it. This allows us to build a common base image for all PHP images. Benefits:

a central place for shared tools and configuration (no more need for a .shared/ directory)
reduced image size when pushing the individual images (the base image is recognized as a layer and thus "already exists")
installing extensions via apk add is a lot faster than via docker-php-ext-install

This new approach has two major downsides:

we depend on the alpine release cycle of PHP (and PHP extensions)
the image build process is more complex, because we must build the base image first before we can build the final images

Fortunately, both issues can be solved rather easily:

codecasts/php-alpine maintains an apk repository with the latest PHP versions for alpine
we use a dedicated make target to build the images instead of invoking docker-compose directly - this enables us to define a "build order" (base first, rest after) while still having to run only a single command as a developer (see Ensuring the build order)

ENV vs ARG

I've noticed that some build arguments are required in multiple PHP containers, e.g. the name of the application user defined in the APP_USER_NAME ENV variable. The username is needed

in the base image to create the user
in the fpm image to define the user that runs the fpm processes (see php-fpm.d/www.conf)
in the worker image to define the user that runs the worker processes ( see supervisor/supervisord.conf)

Instead of passing the name to all images via build argument, i.e.

define it explicitly under services.*.build.args in the docker-compose.yml file
"retrieve" it in the Dockerfile via ARG APP_USER_NAME

I've opted to make the username available as an ENV variable in the base image via

ARG APP_USER_NAME
ENV APP_USER_NAME=${APP_USER_NAME}

and thus be able to access it in the child images directly, I can now write

RUN echo ${APP_USER_NAME}

instead of

ARG APP_USER_NAME
RUN echo ${APP_USER_NAME}

I'm not 100% certain that I like this approach as I'm more or less "abusing" ENV variables in ways that they are likely not intended ("Why would the username need to be stored as an ENV variable?") -
but I also don't see any other practical downside yet.

Image naming convention

Defining a fully qualified name for images will make it much easier to reference the images later, e.g. when pushing them to the registry.

The naming convention for the images is $(DOCKER_REGISTRY)/$(DOCKER_NAMESPACE)/$(DOCKER_SERVICE_NAME)-$(ENV), e.g.

                   docker.io/dofroscra/nginx-local
$(DOCKER_REGISTRY)---^          ^        ^     ^        docker.io
$(DOCKER_NAMESPACE)-------------^        ^     ^        dofroscra
$(DOCKER_SERVICE_NAME)-------------------^     ^        nginx
$(ENV)-----------------------------------------^        local

and it is used as value for services.*.image, e.g. for nginx

services:
  nginx:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/nginx-${ENV?}:${TAG?}

In case you are wondering: dofroscra stems from Docker From Scratch

Environments and build targets

Our final goal is a setup that we can use for

local development
in a CI/CD pipeline
in production

and even though we strive to for a parity between those different environments, there will be differences due to fundamentally different requirements. E.g.

on production I want a container including the sourcecode without any test dependencies
on CI I want a container including the sourcecode WITH test dependencies
on local I want a container that mounts the sourcecode from my host (including dependencies)

This is reflected through the ENV environment variable. We use it in two places:

as part of the image name as a suffix of the service name (see Image naming convention) 2. to specify the target build stage

See the docker-compose-php-base.yml file for example:

services:
  php-base:
    image: ${DOCKER_REGISTRY?}/${DOCKER_NAMESPACE?}/php-base-${ENV?}:${TAG?}
    build:
      dockerfile: images/php/base/Dockerfile
      target: ${ENV?}

Using multiple targets in the same Dockerfile enables us to keep a common base but also include environment specific instructions. See the Dockerfile of the php-base image for example

ARG ALPINE_VERSION
FROM composer:${COMPOSER_VERSION} as composer
FROM alpine:${ALPINE_VERSION} as base

RUN apk add --update --no-cache \
        bash

WORKDIR $APP_CODE_PATH

FROM base as local

RUN apk add --no-cache --update \
        mysql-client \

it first defines a base stage that includes software required in all environments
and then defines a local stage that adds additionally a mysql-client that helps us to debug connectivity issues

After the build for local is finished, we end up with an image called php-base-local that used the local build stage as target build stage.

Makefile

In the following section I will introduce a couple of commands, e.g. for building and running containers. And to be honest, I find it kinda challenging to keep them in mind without having to look up the exact options and arguments. I would usually create a helper function or an alias in my local .bashrc file in a situation like that - but that wouldn't be available to other members of the team then and it would be very specific to this one project.

Instead we'll use a self-documenting Makefile that acts as the central entrypoint in the application. Since Makefiles tend to grow over time, I've adopted some strategies to keep them "sane" via includes, shared variables and better error handling.

`.make/*.mk` includes

Over time the make setup will grow substantially, thus we split it into multiple .mk files in the .make/ directory. The individual files are prefixed with a number to ensure their order when we include them in the main Makefile via

include .make/*.mk

.
└── .make/
    ├── 01-00-application-setup.mk
    ├── 01-01-application-commands.mk
    └── 02-00-docker.mk

Shared variables: `.make/.env`

We try to make shared variables available here, because we can then pass them on to individual commands as a prefix, e.g.

.PHONY: some-target
some-target: ## Run some target
    ENV_FOO=BAR some_command --baz

This will make the ENV_FOO available as environment variable to some_command.

Shared variables are used by different components, and we always try to maintain only a single source of truth. An example would be the DOCKER_REGISTRY variable that we need to define the image names of our docker images in the docker-compose.yml files but also when pushing/pulling/deploying images via make targets later. In this case, the variable is required by make as well as docker-compose.

To have a clear separation between variables and "code", we use a .env file located at . make/.env. It can be initialized via

make make-init

by copying the .make/.env.example to .make/.env.

.
└── .make/
    ├── .make/.env.example
    └── .make/.env

The file is included in the main Makefile via

-include .make/.env

The - prefix ensures that make doesn't fail if the file does not exist (yet), see GNU make: Including Other Makefiles.

Manual modifications

You can always modify the .make/.env file manually if required. This might be the case when you run docker on Linux and need to match the user id of your host system with the user id of the docker container. It is common that your local user and group have the id 1000. In this case you would add the entries manually to the .make/.env file.

APP_USER_ID=1000
APP_GROUP_ID=1000

See also section Solving permission issues.

Enforce required parameters

We kinda "abuse" make for executing arbitrary commands (instead of building artifacts) and some of those commands require parameters that can be passed as command arguments in the form

make some-target FOO=bar

There is no way to "define" those parameters as we would in a method signature - but we can still ensure to fail as early as possible if a parameter is missing via

@$(if $(FOO),,$(error FOO is empty or undefined))

We use this technique for example to ensure that all required variables are defined when we execute docker targets via the validate-docker-variables precondition target:

.PHONY: validate-docker-variables
validate-docker-variables: 
    @$(if $(TAG),,$(error TAG is undefined))
    @$(if $(ENV),,$(error ENV is undefined))
    @$(if $(DOCKER_REGISTRY),,$(error DOCKER_REGISTRY is undefined - Did you run make-init?))
    @$(if $(DOCKER_NAMESPACE),,$(error DOCKER_NAMESPACE is undefined - Did you run make-init?))
    @$(if $(APP_USER_NAME),,$(error APP_USER_NAME is undefined - Did you run make-init?))
    @$(if $(APP_GROUP_NAME),,$(error APP_GROUP_NAME is undefined - Did you run make-init?))

.PHONY:docker-build-image
docker-build-image: validate-docker-variables
    $(DOCKER_COMPOSE) build $(DOCKER_SERVICE_NAME)

Make + Docker = <3

We already introduced quite some complexity into our setup:

"global" variables (shared between make and docker)
multiple docker-compose.yml files
build dependencies

Bringing it all together "manually" is quite an effort and prone to errors. But we can nicely tuck the complexity away in .make/02-00-docker.mk by defining the two variables DOCKER_COMPOSE and DOCKER_COMPOSE_PHP_BASE

DOCKER_DIR:=./.docker
DOCKER_ENV_FILE:=$(DOCKER_DIR)/.env
DOCKER_COMPOSE_DIR:=$(DOCKER_DIR)/docker-compose
DOCKER_COMPOSE_FILE:=$(DOCKER_COMPOSE_DIR)/docker-compose.yml
DOCKER_COMPOSE_FILE_LOCAL:=$(DOCKER_COMPOSE_DIR)/docker-compose.local.yml
DOCKER_COMPOSE_FILE_PHP_BASE:=$(DOCKER_COMPOSE_DIR)/docker-compose-php-base.yml
DOCKER_COMPOSE_PROJECT_NAME:=dofroscra_$(ENV)

DOCKER_COMPOSE_COMMAND:=ENV=$(ENV) \
 TAG=$(TAG) \
 DOCKER_REGISTRY=$(DOCKER_REGISTRY) \
 DOCKER_NAMESPACE=$(DOCKER_NAMESPACE) \
 APP_USER_NAME=$(APP_USER_NAME) \
 APP_GROUP_NAME=$(APP_GROUP_NAME) \
 docker-compose -p $(DOCKER_COMPOSE_PROJECT_NAME) --env-file $(DOCKER_ENV_FILE)

DOCKER_COMPOSE:=$(DOCKER_COMPOSE_COMMAND) -f $(DOCKER_COMPOSE_FILE) -f $(DOCKER_COMPOSE_FILE_LOCAL)
DOCKER_COMPOSE_PHP_BASE:=$(DOCKER_COMPOSE_COMMAND) -f $(DOCKER_COMPOSE_FILE_PHP_BASE)

DOCKER_COMPOSE uses docker-compose.yml and extends it with docker-compose.local.yml
DOCKER_COMPOSE_PHP_BASE uses only docker-compose-php-base.yml

The variables can then be used later in make recipes.

Ensuring the build order

As mentioned under PHP images, we need to build images in a certain order and use the following make targets:

.PHONY: docker-build-image
docker-build-image: ## Build all docker images OR a specific image by providing the service name via: make docker-build DOCKER_SERVICE_NAME=<service>
    $(DOCKER_COMPOSE) build $(DOCKER_SERVICE_NAME)

.PHONY: docker-build-php
docker-build-php: ## Build the php base image
    $(DOCKER_COMPOSE_PHP_BASE) build $(DOCKER_SERVICE_NAME_PHP_BASE)

.PHONY: docker-build
docker-build: docker-build-php docker-build-image ## Build the php image and then all other docker images

As a developer, I can now simply run make docker-build - which will first build the php-base image via docker-build-php and then build all the remaining images via docker-build-image (by not specifying the DOCKER_SERVICE_NAME variable, docker-compose will build all services listed in the docker-compose.yml files).

I would argue that the make recipes themselves are quite readable and easy to understand but when we run them with the -n option to only "Print the recipe that would be executed, but not execute it", we get a feeling for the complexity:

$ make docker-build -n
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose-php-base.yml build php-base
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml build

Run commands in the docker containers

Tooling is an important part in the development workflow. This includes things like linters, static analyzers and testing tools but also "custom" tools geared towards your specific workflow. Those tools usually require a PHP runtime. For now, we only have a single "tool" defined in the file setup.php. It ensures that a table called jobs is created.

To run this tool, we must first start the docker setup via make docker-up and then execute the script in the application container. The corresponding target is defined in .make/01-00-application-setup.mk:

.PHONY: setup-db
setup-db: ## Setup the DB tables
    $(EXECUTE_IN_APPLICATION_CONTAINER) php setup.php $(ARGS);

which essentially translates to

docker-compose exec -T --user application application php setup.php

if we are outside of a container and to

php setup.php

if we are inside a container. That's quite handy, because we can run the tooling directly from the host system without having to log into a container.

The "magic" happens in the EXECUTE_IN_APPLICATION_CONTAINER variable that is defined in .make/02-00-docker.mk as

EXECUTE_IN_WORKER_CONTAINER?=
EXECUTE_IN_APPLICATION_CONTAINER?=

EXECUTE_IN_CONTAINER?=
ifndef EXECUTE_IN_CONTAINER
    # check if 'make' is executed in a docker container, 
 # see https://stackoverflow.com/a/25518538/413531
 # `wildcard $file` checks if $file exists, 
 # see https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html
 # i.e. if the result is "empty" then $file does NOT exist 
 # => we are NOT in a container
 ifeq ("$(wildcard /.dockerenv)","")
        EXECUTE_IN_CONTAINER=true
    endif
endif
ifeq ($(EXECUTE_IN_CONTAINER),true)
    EXECUTE_IN_APPLICATION_CONTAINER:=$(DOCKER_COMPOSE) exec -T --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME_APPLICATION)
    EXECUTE_IN_WORKER_CONTAINER:=$(DOCKER_COMPOSE) exec -T --user $(APP_USER_NAME) $(DOCKER_SERVICE_NAME_PHP_WORKER)
endif

We can take a look via -n again to see the resolved recipe on the host system

pascal.landau:/c/_codebase/dofroscra# make setup-db ARGS=--drop -n
ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application application php setup.php --drop

Within a container it looks like this:

root:/var/www/app# make setup-db ARGS=--drop -n
php setup.php --drop;

Solving permission issues

If you are using Linux, you might run into permission issues when modifying files that are shared between the host system and the docker containers when the user id is not the same as explained in section Synchronizing file and folder ownership on shared volumes of the previous tutorial.

In this case, you need to modify the .make/.env manually and add the APP_USER_ID and APP_GROUP_ID variables according to your local setup. This must be done before building the images to ensure that the correct user id is used in the images.

In very rare cases it can lead to problems, because your local ids will already exist in the docker containers. I've personally never run into this problem, but you can read about it in more detail at Docker and the host filesystem owner matching problem. The author even proposes a general solution via the Github project "FooBarWidget/matchhostfsowner".

PHP POC

To ensure that everything works as expected, the repository contains a minimal PHP proof of concept. By default, port 80 from the host ist forwarded to port 80 of the nginx container.

FYI: I would also recommend to add the following entry in the hosts file on the host machine

127.0.0.1 app.local

so that we can access the application via http://app.local instead of http://127.0.0.1.

The files of the POC essentially ensure that the container connections outlined in Local docker setup work as expected:

dependencies.php
- returns configured Redis and PDO objects to talk to the queue and the database
setup.php
- => ensures that application can talk to mysql
public/index.php
- is the web root file that can be accessed via http://app.local
  - => ensures that nginx and php-fpm are working
- contains 3 different "routes":
  - http://app.local?dispatch=some-job-id
    - dispatches a new "job" with the id some-job-id on the queue to be picked up by a worker
      - => ensures that php-fpm can talk to redis
  - http://app.local?queue
    - shows the content of the queue
  - http://app.local?db
    - shows the content of the database
      - => ensures that php-fpm can talk to mysql
worker.php
- is started as daemon process in the php-worker container
- checks the redis datasbase 0 for the key "queue" every second
- if a value is found it is stored in the jobs table of the database
  - => ensures that php-worker can talk to redis and mysql

A full test scenario is defined in test.sh and looks like this:

$ bash test.sh


  Building the docker setup


//...


  Starting the docker setup


//...


  Clearing DB


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application application php setup.php --drop;
Dropping table 'jobs'
Done
Creating table 'jobs'
Done


  Stopping workers


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker supervisorctl stop worker:*;
worker:worker_00: stopped
worker:worker_01: stopped
worker:worker_02: stopped
worker:worker_03: stopped


  Ensuring that queue and db are empty


Items in queue
array(0) {
}
Items in db
array(0) {
}


  Dispatching a job 'foo'


Adding item 'foo' to queue


  Asserting the job 'foo' is on the queue


Items in queue
array(1) {
  [0]=>
  string(3) "foo"
}


  Starting the workers


ENV=local TAG=latest DOCKER_REGISTRY=docker.io DOCKER_NAMESPACE=dofroscra APP_USER_NAME=application APP_GROUP_NAME=application docker-compose -p dofroscra_local --env-file ./.docker/.env -f ./.docker/docker-compose/docker-compose.yml -f ./.docker/docker-compose/docker-compose.local.yml exec -T --user application php-worker supervisorctl start worker:*;
worker:worker_00: started
worker:worker_01: started
worker:worker_02: started
worker:worker_03: started


  Asserting the queue is now empty


Items in queue
array(0) {
}


  Asserting the db now contains the job 'foo'


Items in db
array(1) {
  [0]=>
  string(3) "foo"
}

Wrapping up

Congratulations, you made it! If some things are not completely clear by now, don't hesitate to leave a comment. Apart from that, you should now have a running docker setup and the means to "control" it conveniently via make.

In the next part of this tutorial, we will configure PhpStorm as our IDE to use the docker setup.

Please subscribe to the RSS feed or via email to get automatic notifications when this next part comes out :)

DEV Community: Pascal Landau

A primer on GCP Compute Instance VMs for dockerized Apps [Tutorial Part 8]

Table of contents

Introduction

Set up a GCP project

Create a service account

Create service account key file

Configure IAM permissions

Set up the gcloud CLI tool

Set up the Container Registry

Authenticate docker

Pushing images to the registry

Images are stored in Google Cloud Storage buckets

Pulling images from the registry

Set up the Secret Manager

Create a secret via the UI

View a secret via the UI

Retrieve a secret via the gcloud cli

Add the secret gpg key and password

Compute Instances: The GCP VMs

Create a VM

General VM settings

Firewall and networks tags

The role of the service account

Adding a public SSH key

Define Availability Policies

The actual VM creation

Log into a VM

Login via SSH from the GCP UI

Login via SSH with your own key from your host machine

Login using the Identity-Aware Proxy (IAP) concept

Additional notes on IAP

Get root permissions

ssh and scp commands

gcloud ssh --command=""

gcloud scp

Provision the VM

Get the secret gpg key and password from the Secret Manager

Installing docker and docker compose

Authenticate docker via gcloud

Pulling the nginx image

Start the nginx container

Automate via gcloud commands

Preconditions: Project and Owner service account

Configure gcloud to use the master service account

Enable APIs

Create and configure a "deployment" service account

Create secrets

Create firewall rule for HTTP traffic

Create a Compute Instance VM

Provisioning

Deployment

Putting it all together

Cleanup

Wrapping up

CI Pipelines for dockerized PHP Apps with Github & Gitlab [Tutorial Part 7]

Table of contents

Introduction

Recommended reading

Approach

Try it yourself

CI setup

General CI notes

Initialize make for CI

wait-for-service.sh

Setup for a "local" CI run

Run details

Execution example

Setup for Github Actions

The Workflow file

Setup for Gitlab Pipelines

The .gitlab-ci.yml pipeline file

Performance

The caching problem on CI

Docker changes

Compose file updates

docker-compose.local.yml

docker-compose.ci.yml

Adding a health check for mysql

Build target: ci

Set up the `gcloud` CLI tool

Retrieve a secret via the `gcloud` cli

Add the secret `gpg` key and password

Get `root` permissions

`ssh` and `scp` commands

`gcloud ssh --command=""`

`gcloud scp`

Get the secret `gpg` key and password from the Secret Manager

Installing `docker` and `docker compose`

Authenticate docker via `gcloud`

Pulling the `nginx` image

Start the `nginx` container

Automate via `gcloud` commands

Preconditions: Project and `Owner` service account

Configure `gcloud` to use the master service account

Initialize `make` for CI

The `.gitlab-ci.yml` pipeline file

Adding a health check for `mysql`

Build target: `ci`

Build stage `ci` in the `php-base` image

Build stage `ci` in the `application` image

ENV based `docker compose` config

Add a password-protected secret `gpg` key

How to install `make`

Configuration via `.bashrc`

The role of `winpty`: Fixing "The input device is not a TTY"

Fixing the path conversion issue for `winpty`

Change the `bash` custom prompt to a `$`

The `git` permission issue

The `git-secret` directory and the `gpg-agent` socket

Local `git-secret` and `gpg` setup

Initial setup of `gpg` keys

Initial setup of `git-secret`

Initialize `gpg` after container startup