Building docker images in docker, and dynamic sequential Jenkins stages

Jonas Bergström — Thu, 17 Feb 2022 09:51:29 +0000

A couple of things I thought would be super easy but turned out to require a few hours of research and trial-and-error...

At GOALS we run Jenkins in Kubernetes for various reasons.
Some build jobs generates Docker images as artefacts, for example when we build new versions of some backend service.
However, Kubernetes is deprecating Docker, and we are running all our nodes on containerd.
So, how to build Docker images then?

Turns out that Google has a project for it, named Kaniko. And since we're running GKE, and has Workload Identity properly configured, it should be super easy to build Docker images and push them to our GCP managed docker repo, right?

No. I get this error:

aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "europe-west1-docker.pkg.dev/XXX": creating push check transport for europe-west1-docker.pkg.dev failed: GET https://europe-west1-docker.pkg.dev/v2/token?YYY: UNAUTHORIZED: authentication failed

AWS credentials - wth?

After some testing I found this, which solved the issue :). For an example Jenkinsfile, see below.

Another issue I had was that I wanted to generate dynamic build stages in Jenkins, and execute them sequentially. There are a lot of examples of how to execute dynamically generated stages in parallel, but that's not what I wanted.
Turned out to be super simple in the end ofc, but it aint simple until you've learned.

Here's an example Jenkinsfile that demonstrates both Kaniko and dynamic sequential build steps:

def stageConfigs = [
  [name: "Sara", age: 20, color: "blue"],
  [name: "Mona", age: 10, color: "green"],
  [name: "Lotta", age: 8, color: "red"]
]

def generateBuildStage(stageConfig) {
  return {
    stage("Building ${stageConfig.name}") {
      container('build-container') {
        echo "Building ${stageConfig.name}"
        writeFile(
          file: "${stageConfig.name}.generated",
          text: "${stageConfig.name} is ${stageConfig.age} years old and loves ${stageConfig.color} things")
      }
    }
  }
}

def buildStages = stageConfigs.collectEntries {
    ["${it}" : generateBuildStage(it)]
}

podTemplate(
  inheritFrom: 'linux',
  containers: [
    containerTemplate(name: "build-container", image: "busybox", command: "sleep", args: "infinity"),
    containerTemplate(name: "kaniko", image: "gcr.io/kaniko-project/executor:343f78408c891ef7a85bab1ecbf2dd69367a58bc-debug", command: "sleep", args: "infinity", runAsUser: "0", ttyEnabled: true)])
  {
  node(POD_LABEL) {
    stage("Checkout") {
      checkout(scm)
    }

    stage("Build application") {
      // execute builds in parallel
      // parallel(buildStages)
      // execute builds sequentially
      for (stage in buildStages.values()) {
        stage.call()
      }
      container('build-container') {
        writeFile(
          file: "Dockerfile",
          text: '''
            FROM busybox
            COPY *.generated ./
          ''')
        sh "ls -al"
      }
    }

    stage('Build image') {
      container('kaniko') {
        sh "/kaniko/executor --context `pwd` --dockerfile `pwd`/Dockerfile --destination europe-west1-docker.pkg.dev/XXX"
      }
    }
  }
}

HOWTO; Connect to a private GKE cluster using a site-to-site VPN between a GCP VPC...

Jonas Bergström — Sat, 12 Feb 2022 16:50:05 +0000

... and a double NAT'ed Ubiquiti Dream Machine Pro

Wait, what? I just want to access a Kubernetes cluster in a secure way ... should be easy.

[2022-03-10] Update from Google! I just got this message: "We are writing to let you know that starting June 15, 2022, we will remove the restriction on the Internet Key Exchange (IKE) identity of peer Cloud VPN gateways."
This means in practice that NAT'ed setups will become easier to manage because GCP will not require remote IP to match remote id.

At GOALS we are cloud native, and we are serious about security. As a consequence our Kubernetes clusters are provisioned with private endpoints only, where all nodes have internal IP addresses. This is great, but the question immediately arrises - how do we operate such a cluster, when we cannot access it from outside the VPC?

Here is a high level overview of what we have:

At the top of the diagram we see the private Google managed Kubernetes (GKE) cluster. A Kubernetes cluster consists of a control plane and worker nodes. In the case of GKE, Google manages the control plane (api server, etcd nodes, etc), the underlying VM's the control plane is running on, and the underlying VM's that worker nodes are running on.
We have set up our own VPC and a subnet where the worker nodes are running, and Google creates a managed VPC where the control plane is running. Google automatically peers the control plane VPC with our VPC.

At the bottom of the diagram we see an overview of our office space network. Since we are a startup we rent office space and share the network with other tenants. We have connected a Ubiquity Dream Machine Pro to the office space network, and created our own GOALS network where we connect our work stations.

Obviously, running eg kubectl describe nodes from my workstation in our office network doesn't work since kubectl needs access to the cluster's api server. So, how can we connect our office network to our VPC in a secure way, and enable management of the GKE cluster using kubectl?

A note on our infrastructure

We use Terraform to provision all GCP resources. Google provides opinionated Terraform modules to manage GCP resources here.
Our infra leverages a shared VPC and we use the project factory module to create the host project and the service projects.
The VPN will be provisioned in the host project that owns the VPC.
Our GKE clusters are created with the private cluster terraform module.

Prerequisites and preparations

There are a few things we need to ensure and some information we must gather before we can start.
First, we need admin access to the Ubiquiti Dream Machine (UDM) and we need a Google user with Network Management Admin.
Now you need to gather the following:

The office space external IP address, eg 123.45.67.89
The Goals network subnet range, eg 192.168.1.0/24

Ensure that the Kubernetes cluster is prepared

Two configuration entries are necessary to get right when setting up GKE to enable VPN access to the cluster.

master_authorized_networks needs to include the office network subnet range, ie 192.168.1.0/24 in our case.
VPC peering must be configured to export custom routes - in this particular case it means that the custom network route that the VPN will create to enable communication from the VPC to the office network will also be available in the Google managed VPC that host the GKE control plane. This is enabled by adding the following terraform configuration to the GKE setup: ```

module "kubernetes_cluster" {
source = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
version = "18.0.0"
...
}

resource "google_compute_network_peering_routes_config" "peering_gke_routes" {
peering = module.kubernetes_cluster.peering_name
network = var.vpc_id
import_custom_routes = false
export_custom_routes = true
}


### Shared VPN secret
Generate a shared secret that will be used for authentication of the VPN peers and put it into a new secret in GCP Secret Manager, name it **office-vpn-shared-secret**.

### Create a Cloud VPN Gateway
For this part we will use the [VPN](https://github.com/terraform-google-modules/terraform-google-vpn) module, setting it up in the host project like this:

locals {
name = "office-vpn"
}

resource "google_compute_address" "vpn_external_ip_address" {
project = var.project_id
name = local.name
network_tier = "PREMIUM"
region = var.region
address_type = "EXTERNAL"
}

data "google_secret_manager_secret_version" "office_vpn_shared_secret" {
project = var.project_id
secret = "office-vpn-shared-secret"
}

module "office_site_to_site" {
source = "terraform-google-modules/vpn/google"
version = "2.2.0"

project_id = var.project_id
network = var.vpc_id
region = var.region
gateway_name = local.name
tunnel_name_prefix = local.name
shared_secret = data.google_secret_manager_secret_version.office_vpn_shared_secret.secret_data
ike_version = 2
peer_ips = [ var.office_public_ip ]
remote_subnet = var.office_subnet_ranges
vpn_gw_ip = resource.google_compute_address.vpn_external_ip_address.address
}

resource "google_compute_firewall" "allow_office_traffic" {
project = var.project_id
name = "${local.name}-allow-office-traffic"
network = var.vpc_id
description = "Allow traffic from the office network"
allow { protocol = "icmp" }
allow {
protocol = "udp"
ports = [ "0-65535" ]
}
allow {
protocol = "tcp"
ports = [ "0-65535" ]
}
source_ranges = var.office_subnet_ranges
}

`google_compute_address.vpn_external_ip_address` creates the external static IP address which becomes the VPN endpoint on the GCP end.
`google_secret_manager_secret_version.office_vpn_shared_secret` fetches the shared secret used to authenticate the VPN peers.
The `office_site_to_site` module creates a "classic" Cloud VPN. The UDM does not support BGP yet which means that we cannot create a "HA Cloud VPN" variant.
`google_compute_firewall.allow_office_traffic` allows traffic originating from the office subnet (`192.168.1.0/24`) to enter our VPC.

After applying the configuration the VPN tunnel will be in an error state because it cannot connect to it's peer. This is expected since we have not set up the UDM side yet.

### Create a VPN network on the UDM Pro
The UDM Pro does not support configuration by code as far as I know, so here we need to resort to manually use the management GUI.

Go to Settings->Networks->Add New Network, choose a name and select VPN->Advanced->Site-to-site->Manual IPSec.
**Pre-shared Secret Key** is the `office-vpn-shared-secret` from above.
**Public IP Address (WAN)** is the IP address the UDM has on the office space network, ie it is __not__ the public IP our office space provider has. For example `192.168.10.150`.
In the **Remote Gateway/Subnets** section, add the subnet ranges in your VPC that you want to access from the office, eg `10.0.0.0/8` and `172.16.0.0/16`.
The **Remote IP Address** is the public static IP that was created for the VPN endpoint in GCP, eg `123.45.67.99`.
Expand the Advanced section and choose `IKEv2`. Leave PFS and Dynamic routing enabled.
Save the new network.

Unfortunately we are not ready yet, because with the current configuration the UDM will identify itself using the WAN IP we have configured, which doesn't match the IP it connects to GCP with.
To fix this last part of configuration we need to `ssh` into the UDM Pro. Once you are on the machine, we can update the ipsec configuration.

$ cd /run/strongswan/ipsec.d/tunnels
$ vi .config

Add the following line just below the `left=192.168.10.150` line:

leftid=123.45.67.99

This makes the UDM identify itself using it's actual public IP when it connects to the VPN on the GCP end.
Finally, refresh the IPSec configuration:

$ ipsec update


## Thats it!
Verify that the connection on the UDM is up and running by invoking:

$ swanctl --list-sas

The output should list information about the tunnel, and the tunnel should be in {% raw %}`ESTABLISHED` state.
Now the VPN tunnel state in GCP should move into a green state as well.
And, finally, accessing the Kubernetes cluster from a workstation in the Goals office network is possible:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-XXXXX-controller-pool-28b7a87b-9ff2 Ready 17d v1.21.6-gke.1500

We now have a setup looking like below:

![Private GKE and office with VPN](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zlornjjrpq8ya2eldfkx.png)

## Troubleshooting
While being connected to the UDM, run

$ tcpdump -nnvi vti64

to see all traffic routed via the VPN tunnel.

Sometimes routes are cached on the workstations, eg on Mac you can run

$ sudo route -n flush

a couple of times and disable/enable Wifi to make sure that your routing configuration is up to date.

DEV Community: Jonas Bergström

Building docker images in docker, and dynamic sequential Jenkins stages

HOWTO; Connect to a private GKE cluster using a site-to-site VPN between a GCP VPC...

A note on our infrastructure

Prerequisites and preparations

Ensure that the Kubernetes cluster is prepared