DEV Community: lykins

First Look Nomad 1.11.x - System Job Deployments

lykins — Wed, 14 Jan 2026 15:56:22 +0000

Originally posted : https://blog.lykins.xyz/posts/system-job-deployments/

This last Nomad 1.11.x feature I will be covering is not new to Nomad itself, putting a new twist to how you update and manage system jobs.

What's New?

In Nomad 1.11.x, system jobs now support deployment functionality which will allow operators to manage changes and rollbacks for system jobs more effectively. This enhancement gives additional control over how system jobs are deployed and maintained across your infrastructure.

As always, I would heavily recommend reading the release notes, which I have linked above, and the documentation on the update block as it has been updated to call out different behaviors when used with system jobs, such as the following:

In system jobs, the canary setting indicates the percentage of feasible nodes to which Nomad makes destructive allocation updates. System jobs do not support more than one allocation per node, so effectively setting canary to a positive integer means this percentage of feasible nodes gets a new version of the job if the update is destructive. Non-destructive updates ignore the canary field. Setting canary to 100 updates the job on all nodes. Percentage of nodes is always rounded up to the nearest integer. If canary is set, nodes that register during a deployment do not receive placements until after the deployment is promoted.

Since this also updates the Nomad UI, I will include screenshots as well.

Comparing 1.11.x to Previous Versions

So to start off, let's do a system job run comparison to see how system jobs behaved in previous versions of Nomad versus how they behave in 1.11.x.

I will have one cluster which is running 1.10.5 and another running 1.11.1. Both clusters are mostly identically set up - 1 server and 3 clients each.

Demo Job

Running a very simple system job that deploys a busybox container on all clients in the cluster. Here is the jobspec:

job "system-deployment" {
  datacenters = ["dc1"]
  type        = "system"

  update {
    canary            = 30
    max_parallel      = 1
    min_healthy_time  = "30s"
    healthy_deadline  = "5m"
    auto_promote      = false
    auto_revert       = true
  }

  group "test-group" {
    task "test-task" {
      driver = "docker"

      config {
        image = "busybox:1.36"
        command = "sh"
        args    = ["-c", "echo 'Running...'; sleep 3600"]
      }
    }
  }
}

Pay a bit more attention to the update block and see how it is set up for this demo.

canary is set to 30, when we do a destructive update, it will update 30% of the nodes first (1 of 3 nodes in this case) before promoting the version to the remaining nodes.
max_parallel is set to 1, only one node will be updated at a time.
min_healthy_time is set to 30s, the updated allocation must be healthy for at least 30 seconds before proceeding.
healthy_deadline is set to 5m, if the updated allocation does not become healthy within 5 minutes, the deployment will be considered failed.
auto_promote is set to false, the deployment will not automatically proceed to update the remaining nodes after the canary node is healthy.
auto_revert is set to true, if the deployment fails, it will automatically revert to the previous version.

Canary is not necessary to trigger a deployment, but provides

UI

On the left side of the screen is my homelab running 1.11.1 and on the right side is a quick lab I spun up on multipass with 1.10.5. See if you can spot the subtle differences in the UI for system jobs between the two versions:

In the 1.11.1 UI on the left, you can see the "Deployments" tab exists now for the system job, whereas in 1.10.5 on the right, there is no such tab. With that, you will have additional deployment information and status on the overview page.

Update Behavior

Now let's look at the update behavior between the two versions.

Running an Updated Job

For this, I will update the container image version from 1.36 to 1.37 and run the update in the UI. Since this is a destructive update, you will see how each version handles it differently.:

Nothing different or major to see here, so we will carry along.

Deployment Behaviors

After running the update:

On the left side with 1.11.x, you can see the deployment is in progress, and it is updating one of the three nodes (50% as per the canary setting).

On the right side with 1.10.5, a deployment is not created, and the job is being updated per the max_parallel setting.

Promotion

You can see now with 1.11.x, the deployment is in progress, and it will wait till it is healthy based on any health checks - in this case it is only healthy allocations. Once healthy, it will require manual promotion to continue the deployment to the remaining nodes as auto_promote is set to false as shown below:

Here I have the option, to promote or revert the deployment. If this was a real job, I could do some additional testing or validation, but for this demo, I will go ahead and promote the deployment and Nomad will handle updating the remaining allocations.

Conclusion

System job deployments in Nomad 1.11.x bring a new level of control and reliability to managing critical system services across your infrastructure. By leveraging deployments, manual or controlled promotions, and automatic rollbacks, operators can ensure that updates to system jobs are handled safely and efficiently.

First Look Nomad 1.11.x - Secret Block

lykins — Fri, 19 Dec 2025 17:20:21 +0000

Originally posted: https://blog.lykins.xyz/posts/secret-block/

The next Nomad 1.11.x feature we will be looking at is Artifact Secrets which introduces a new secret block in the job specification to simplify the process of fetching secrets for your Nomad jobs in addition to one key difference compared to the existing template block.

Out of the box there are two built-in plugins which support Nomad variables and Vault. In addition to the built-in plugins there is a secret provider plugin framework which allows community or privately built plugins to support their secrets management systems. For example, check out the demo they provided for AWS Secrets Manager.

Note on my Setup

For this post, portions of it I will be using Vault to manage my secrets. If you do not have Vault set up and would like to spin up a development instance to test with, check out the workload identity tutorial for a quick start.

Before Diving In

I want to make it clear that the new secret block is not a 1 to 1 replacement for the existing template block. They both have their own use cases and could potentially be used together in the same job specification.

The most notable feature with the secret block is that it can be interpreted in your job specification, which the template block could not do. I'll get into more detail on that after this.

When it comes to grabbing secrets to be used in your workloads, the secret block is best suited for single execution workloads such as batch, dispatched, or periodic jobs that need secrets at start time. For longer running workloads that may need to dynamically refresh secrets or other configuration, the template block would still be the recommended as it can more dynamically manage your workloads if there is a change.

For example, if you were to fetch a TLS certificate from Vault using the template block and you want the task to restart when the certificate is renewed, you would do something like this:

template {
  data = <<EOH
{{ with pkiCert "pki/issue/foo" "common_name=foo.service.consul" "ip_sans=127.0.0.1" "format=pem" }}
{{ .Cert }}
{{ .CA }}
{{ .Key }}{{ end }}
EOH
  destination   = "${NOMAD_SECRETS_DIR}/bundle.pem"
  change_mode   = "restart"
}

When and if there is a change, the task will be restarted to pick up the new certificate. This is not something that can be accomplished with the secret block as it is only fetched at start time and cannot trigger restarts on changes.

With that out of the way, lets dive into the new secret block and see how it works.

What Makes the Secret Block Special?

The main advantage of using the secret block is that it can be interpreted in your job specification. This means you can directly reference secrets in your job specification or prestaged configuration in order to authenticate to pull images or artifacts.

To show what this looks like, let's start with a very simple image I built and stored privately in GitHub's container registry and see how Nomad currently acts when attempting to pull without authentication.

job "secret-image" {
  datacenters = ["*"]
  group "http" {
    count = 1
    network {
      port "http" {
        to = 8090
      }
    }
    task "http" {
      driver = "docker"
      config {
        image   = "ghcr.io/benjamin-lykins/nomad-secret-block-demo:latest"
        ports   = ["http"]
      }
    }
  }
}

We get this pleasant message in the UI:

So it looks like we cannot get the image due to authentication issues and this is where the secret block comes into play.

Historically, authentication for the Docker task driver was handled within the auth block in the job's task specification. Which could require some configuration to be staged prior to running the job.

Now we can do it this way:

Make sure you have a valid GitHub personal access token (PAT) with the read:packages scope.

We will keep it easy and store it as a Nomad variable at nomad/jobs - you might want more narrow scoping in your environments, but for testing this is simple enough - with the key github_token. I will also include my GitHub username as a variable at nomad/jobs with the key github_user to easily reference it in the job specification.

Update the job specification to use the new secret block to fetch the token from Vault and use it in the auth block for the Docker driver.

task "http" {
    driver = "docker"
    config {
      image   = "ghcr.io/benjamin-lykins/nomad-secret-block-demo:latest"
      ports   = ["http"]

      auth {
        username = "${secret.github.github_user}"
        password = "${secret.github.github_token}"
        server_address = "ghcr.io"
      }
    }
  }
  secret "github" {
    provider = "nomad"
    path     = "nomad/jobs/"
  }

There are two key takeaways here:

The secret block was added which defines where to fetch the secrets from. In this case, we are using the Nomad variable provider to fetch both the username and token from nomad/jobs/.
Updated my task configuration to reference both the username and password in the auth block using ${secret.github.github_user} and ${secret.github.github_token} syntax.

Running the job now successfully pulls the image from GitHub's container registry. Voilà - up and running!

This provides a much simpler way to manage authentication for pulling images or artifacts in your Nomad jobs.

And although I have only tested this with Docker, it should also simplify authentication for other task drivers that support downloading artifacts and images.

Diving a Bit Deeper

I wanted to play around a bit more and see how the new secret block compares to the existing template block for fetching secrets and using them within your workloads. Say if you have a secret stored in Vault which you want to use in your task as an environment variable.

Fetching Secrets with the Template Block

Previously, the only option was to use was the template block in your job specification to render secrets or configuration from Vault, Nomad, Consul, or local files. This is built with go templating and uses Consul Template under the hood. This was a powerful and flexible way to manage secrets, but it did come with some complexity. Additionally, you are limited to placing this in the task block.

job -> group -> task -> template

Here is an example of using the template block to fetch a secret from Vault and inject it into the task's environment variables:

job "template-block-demo" {
  type = "service"

  datacenters = ["dc1"]
  group "template-block-demo" {
    task "template-block-demo" {
      driver = "exec"
      template {
        destination = "secrets/file.env"
        env         = true
        data = <<EOF
SECRET = "{{ with secret "kv/data/default/template-block-demo" }}{{ .Data.data.secret }}{{ end }}"
EOF
      }

      vault {}

      config {
        command = "/bin/sh"
        args = [
          "-c",
          "while true; do echo \"Secret $SECRET\"; sleep 5; done"
        ]  
      }
    }
  }
}

Verifying this works as expected, we can see the secret being printed out by the task:

So you can see the template block is working and pulling the secret from key secret with a value of Agent Man.

Fetching Secrets with the Secret Block

With Nomad 1.11.x you can now define secrets directly by using the new secret block in your job specification.

This block can also be used in more levels than just the task block. So if you had a common token or secret to pull images or artifacts, you could define it at the job or group level and reference it in multiple tasks.

job -> secret \
job -> group -> secret \
job -> group -> task -> secret

Recreating the previous example using the new secret block. I will define the secret at the job level this time, moving it from the task block, so it can be reused across multiple tasks if needed.

  secret "my_secret" {
    provider = "vault"
    path     = "kv/data/default/secret-block-demo"
    config {
      engine = "kv_v2"
    }
  }

I reference the secret directly into the task's config using ${secret.my_secret.secret} syntax versus templating and injecting it in as an environment variable.

Which will take the task config from this:

      template {
        destination = "secrets/file.env"
        env         = true
        data = <<EOF
SECRET = "{{ with secret "kv/data/default/template-block-demo" }}{{ .Data.data.secret }}{{ end }}"
EOF
      }

      config {
        command = "/bin/sh"
        args = [
          "-c",
          "while true; do echo \"Secret $SECRET\"; sleep 5; done"
        ]  
      }

To know being able to reference it directly like so:

  args = [
    "-c",
    "while true; do echo \"Secret ${secret.my_secret.secret}\"; sleep 5; done"
  ]

Full job specification below:

job "secret-block-demo" {
  type = "service"

  secret "my_secret" {
    provider = "vault"
    path     = "kv/data/default/secret-block-demo"
    config {
      engine = "kv_v2"
    }
  }

  datacenters = ["dc1"]
  group "secret-block-demo" {
    task "secret-block-demo" {
      driver = "exec"

      vault {}

      config {
        command = "/bin/sh"
        args = [
          "-c",
          "while true; do echo \"Secret ${secret.my_secret.secret}\"; sleep 5; done"
        ]  
      }
    }
  }
}

Verifying this works as expected, we can see the same secret being printed out by the task:

Pulling Multiple Secrets with the Secret Block

You can also use multiple secret blocks in a single job specification. In this example, I am pulling one secret from kv/data/default/multi-secret-block-demo and another from kv/data/global-secrets.

My workload identity and role will allow any secret to be pulled from global-secrets, and then any job specific secrets can be pulled from their respective paths dynamically.

In this set up, I define both secrets at the job level.

Then I reference them in the task using ${secret.job_secret.job} and ${secret.global_secret.global}.

job "multi-secret-block-demo" {
  type = "service"
  datacenters = ["dc1"]

  secret "job_secret" {
    provider = "vault"
    path     = "kv/data/default/multi-secret-block-demo"
    config {
      engine = "kv_v2"
    }
  }

  secret "global_secret" {
    provider = "vault"
    path     = "kv/data/global-secrets"
    config {
      engine = "kv_v2"
    }
  }

  group "multi-secret-block-demo" {
    task "multi-secret-block-demo" {
      driver = "exec"

      vault {}

      config {
        command = "/bin/sh"
        args = [
          "-c",
          "while true; do echo \"Job: ${secret.job_secret.job}, Global: ${secret.global_secret.global}\"; sleep 5; done"
        ]
      }
    }
  }
}

Verifying this works as expected, we can see both secrets being printed out by the task:

As shown in the output, both secrets are successfully retrieved and displayed.

Using the Secret Block with Env Block

In my previous two examples, I directly referenced the secret in the task's config block. This last demo I want to show is using the secret block in conjunction with the task's env block to set environment variables for the task. So if you have a workload which checks for the existence of environment variables for authentication or configuration, you can use this method to set them.

For this let's say you have a simple, but repetitive task of uploading processed files from our private datacenter to an S3 bucket that has been running as a cron job on a server for years. The server, appropriately named "DO-NOT-DELETE", has served you well, but it is time to send it on its way to retirement and migrate this workload to Nomad.

So to keep this simple so any future administrator or engineer can understand, we will leverage the AWS cli to upload files to S3. Which we will need to provide our AWS credentials to the task as environment variables.

Here is how we can accomplish this using the secret block to fetch our AWS credentials and set them as environment variables in the task's env block.

Store your AWS credentials in your secrets manager. In my case, I will be using Nomad variables with the path nomad/jobs/s3-uploader and my necessary AWS environment variables.
Create a Nomad job specification that uses the secret block to fetch the AWS credentials and set them as environment variables in the task's env block. I'm going to use raw_exec as the driver to keep it simple and just run a shell command to upload files to S3. The AWS cli is already installed on my Nomad client.

job "s3-uploader" {
  type        = "batch"
  datacenters = ["dc1"]

  secret "aws_credentials" {
    provider = "nomad"
    path     = "nomad/jobs/s3-uploader"
  }

  periodic {
    crons            = ["@hourly"]
    prohibit_overlap = true
  }

  group "s3-uploader" {
    task "s3-uploader" {
      driver = "raw_exec"

      env {
        AWS_ACCESS_KEY_ID     = "${secret.aws_credentials.AWS_ACCESS_KEY_ID}"
        AWS_SECRET_ACCESS_KEY = "${secret.aws_credentials.AWS_SECRET_ACCESS_KEY}"
        AWS_DEFAULT_REGION    = "${secret.aws_credentials.AWS_DEFAULT_REGION}"
      }

      config {
        command = "/bin/sh"
        args = [
          "-c",
          "aws s3 sync /data/processed s3://the-bucket-i-will-use-for-this-nomad-demo/"
        ]
      }
    }
  }
}

For this, I want to focus how I am specifically using the secret block to fetch my AWS credentials from Nomad variables at the path nomad/jobs/s3-uploader. Then in the task's env block, I am setting the necessary AWS environment variables by referencing the secrets using ${secret.aws_credentials.<KEY>} syntax. With that set, I should be able to submit this job and at execution time, it will pull the latest AWS credentials and use them to authenticate with S3.

I'm too impatient so I will launch the job manually instead of waiting for the periodic schedule and verify it works as expected.

upload: /data/processed/1 to s3://the-bucket-i-will-use-for-this-nomad-demo/1
Completed 1 file(s) with 2 file(s) remaining
upload: /data/processed/3 to s3://the-bucket-i-will-use-for-this-nomad-demo/3
Completed 2 file(s) with 1 file(s) remaining
upload: /data/processed/2 to s3://the-bucket-i-will-use-for-this-nomad-demo/2

You could make this a bit more robust and use a poststop task to clean up the folder after the upload, but for time being this works and it would keep your files in sync.

Before I finish writing a book, I will stop here and hope that you find this guide useful for understanding how to use the new secret block in Nomad 1.11.x to fetch secrets for your workloads.

In Summary

For use cases when you need to fetch secrets for authenticating to pull images or artifacts, the new secret block in Nomad 1.11.x simplifies the process by allowing direct references in your job specification, while also providing an alternative to using the template block when appropriate.

When needing to pass a secret into your workloads, both the template and secret blocks have their own use cases and can be used together if needed:

Method	Template Block	Secret Block
Location	task block only	job, group, or task block
Syntax	templating	direct reference
Reusability	limited to task level	dependent on placement in the job
Ease of Use	more robust, with a tradeoff on complexity	simpler and more straightforward
Secret Providers	Vault, Nomad, Consul, local files	built-in Vault and Nomad, extensible via plugins
Use Cases	complex templating needs	simple secret retrieval
Workloads	longer running or dynamic workloads	single execution, batch, dispatched, periodic

First Look Nomad 1.11.x - Client Node Introduction and Identity

lykins — Thu, 18 Dec 2025 17:39:44 +0000

Orginally posted here : https://blog.lykins.xyz/posts/nomad-client-introduction/

Nomad 1.11.x was recently released and I wanted to highlight some of the new features in a series of posts.

Client node introduction and identity
Artifact secrets
System job deployments

For this first post, I want to focus on the client node introduction and identity. This will require reading through the release note documentation, updated pages, CLI and API references, as a lot has been added around this feature and I will not cover in detail.

Client Node Introduction

This feature adds in an additional layer of security to prevent unauthorized client from joining a Nomad cluster. Prior to this the first measure of security was valid TLS certificates between the client and server. With this new feature, when strictly enforced, clients must have a valid token to join the cluster.

Think of this as multifactor authentication, but for your Nomad clusters.

Networking - can the client reach the server
TLS - does the client have valid certificates for the cluster
Client Introduction Token - does the client have a valid token to join the cluster

Although I do not cover it in detail, this gives an added benefit in additional control over misconfigured clients trying to join the cluster. You can specify node names, node pools, and TTLs for the tokens you generate.

Nomad Version

This feature requires 1.11.x on both the client and server side to function.

nomad -v
Nomad v1.11.0
BuildDate 2025-11-11T16:18:19Z
Revision 9103d938133311b2da905858801f0e111a2df0a1

Server Configuration

On the server side, a new configuration block has been added to the server configuration. This block is called client_introduction with configurable options.

On my setup, it is going to be very simple as this is a lab environment - so a bit of precaution it might not be up to production standards.

I set my enforcement to strict so that any client that does not have a valid token will be rejected from joining the cluster, instead of a warning.

data_dir = "/opt/nomad/"

acl {
  enabled = true
}

server {
  enabled          = true
  bootstrap_expect = 1
  client_introduction {
    enforcement          = "strict" #Default = "warn"
    default_identity_ttl = "5m"     #Default = "5m"
    max_identity_ttl     = "30m"    #Default = "30m"
  }
}

tls {
  http = true
  rpc  = true
  ca_file   = "/opt/nomad/tls/nomad-agent-ca.pem"
  cert_file = "/opt/nomad/tls/global-server-nomad.pem"
  key_file  = "/opt/nomad/tls/global-server-nomad-key.pem"
}

Starting up my Nomad server with this configuration:

==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.39.125:4646; RPC: 192.168.39.125:4647; Serf: 192.168.39.125:4648
            Bind Addrs: HTTP: [0.0.0.0:4646]; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
                Client: false
             Log Level: INFO
               Node Id: cfc63efc-0f2d-6e60-c27a-4124a9cf8137
                Region: global (DC: dc1)
                Server: true
               Version: 1.11.0

==> Nomad agent started! Log data will stream in below:
...
    2025-11-18T18:32:03.142Z [INFO]  nomad.raft: entering leader state: leader="Node at 192.168.39.125:4647 [Leader]"
    2025-11-18T18:32:03.144Z [INFO]  nomad: cluster leadership acquired
    2025-11-18T18:32:03.162Z [INFO]  nomad.core: established cluster id: cluster_id=bcdaffcb-a79a-d57c-03e4-6cf32357dee5 create_time=1763490723160485342
    2025-11-18T18:32:03.162Z [INFO]  nomad: eval broker status modified: paused=false
    2025-11-18T18:32:03.162Z [INFO]  nomad: blocked evals status modified: paused=false
    2025-11-18T18:32:03.237Z [INFO]  nomad.keyring: initialized keyring: id=9cd82611-3df2-1c73-d010-05be16d750e8

Pretty standard at this point, nothing new outside of the new configuration block. After starting up I bootstrapped the ACL system and pulled the initial management token.

Client Configuration

On the client side, config will stay the same, no additional configuration is needed.Lets set up the Nomad client configuration. Which we should be good to join since we have valid certificates. Once we have the client config set up, we can start the Nomad agent.

data_dir = "/opt/nomad/"

 client {
   enabled = true
   servers = ["192.168.39.125"]
 }

tls {
  http = true
  rpc  = true
  ca_file   = "/opt/nomad/tls/nomad-agent-ca.pem"
  cert_file = "/opt/nomad/tls/global-client-nomad.pem"
  key_file  = "/opt/nomad/tls/global-client-nomad-key.pem"
}

Starting up the Nomad client agent:

==> Loaded configuration from /etc/nomad.d/nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.39.133:4646
            Bind Addrs: HTTP: [0.0.0.0:4646]
                Client: true
             Log Level: INFO
                Region: global (DC: dc1)
                Server: false
               Version: 1.11.0

==> Nomad agent started! Log data will stream in below:
...
    2025-11-18T19:14:17.242Z [INFO]  client: started client: node_id=01f47f9a-7281-c450-6db0-3220fdc016a0
    2025-11-18T19:14:17.250Z [ERROR] client.rpc: error performing RPC to server: error="rpc error: Permission denied" rpc=Node.Register server=192.168.39.125:4647
    2025-11-18T19:14:17.251Z [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: Permission denied" rpc=Node.Register server=192.168.39.125:4647
    2025-11-18T19:14:17.253Z [ERROR] client: error registering: error="rpc error: Permission denied"

Hmm. That did not seem to work.

Monitoring Client Join Failures

Before we dive into how to get the client to join, as platform owners we should be aware of when client nodes are failing to join, so lets go to the server logs.

2025-11-18T19:14:24.952Z [ERROR] nomad.client: node registration without introduction token: enforcement_level=strict node_id=01f47f9a-7281-c450-6db0-3220fdc016a0 node_pool=default node_name=client

From what we can see here, a node attempted to register without an introduction token. If the enforcement was warn, it would join, but it would log as a warning. Since we set it to strict, it is rejected from joining the cluster. For production enviornments, I would recommend after upgrading to 1.11.x to set enforcement to warn first, monitor logs for any unauthorized clients trying to join, and then once you have a repeatble workflow in place, switch to strict enforcement.

Also, if you have telemetry enabled, as you probably should, there are new metrics around client introductions.

How to Generate Client Introduction Tokens

Before we generate our client introduction token, I just want to cover what I have set up for this lab environment:

Got a server
Got a client
Both running 1.11.x
Both with valid TLS certificates - not necessary for client introductions but I guess I've gotten use to always having TLS enabled.
ACL Enabled - please just always make this a habit on all clusters.
- node:write policy is required to generate tokens.
Server has client_introduction block configured.

So our next step is to generate a client introduction token. This can be done via the CLI or API. I will show both ways.

To do this I'm going to walk thorugh creating a policy, role, and then generating a token from that role. We will use that token to request a client introduction token when starting up the Nomad client agent.

Create ACL Policy

# client-introduction.hcl
node {
  policy = "write"
}

nomad acl policy apply client-introduction client-introduction.hcl

Create ACL Role from Policy

This is optional as you can generate tokens directly from policies, but I use roles to help manage my policies better.

nomad acl role create --tls-skip-verify -name="client-introduction" -policy="client-introduction"
ID           = cf0b4a43-b00f-cc30-b656-b34d66151b04
Name         = client-introduction
Description  = <none>
Policies     = client-introduction
Create Index = 117
Modify Index = 117

Generate Nomad Token via CLI

Now we can generate a Nomad token from the role we just created. This token will be used to request client introduction tokens. Ecosystem wise I would recommend Vault to generate and manage these tokens, but for this lab we will just generate it via the CLI.

root@server:~# nomad acl token create -name="client-intro-token-1" -role-name="client-introduction" -type=client -ttl=8h
Accessor ID  = 8c22a7c0-44f5-044d-ef84-bfa06118faf4
Secret ID    = d99d678d-426c-330e-74f3-de53a868e2f9
Name         = client-intro-token-1
Type         = client
Global       = false
Create Time  = 2025-11-18 21:01:44.62075624 +0000 UTC
Expiry Time  = 2025-11-19 05:01:44.62075624 +0000 UTC
Create Index = 128
Modify Index = 128
Policies     = []

Roles
ID                                    Name
cf0b4a43-b00f-cc30-b656-b34d66151b04  client-introduction

Generate Nomad Client Introduction Token via API

This feature introduces a new API endpoint to generate client introduction tokens.

Optional: You can create a JSON file to hold the request body, but in this case we are not any additional parameters so we can just send an empty JSON object.

Example payload file, just to show you can send additional parameters for added control such as NodeName, NodePool, and TTL.:

{
  "NodeName": "node-338ef6e9",
  "NodePool": "platform",
  "TTL": "15m"
}

Curl request to generate client introduction token - I am sending an empty JSON object as the body.:

curl \
    --request POST \
    --header "X-Nomad-Token: d99d678d-426c-330e-74f3-de53a868e2f9" \
    --data {} \
    https://localhost:4646/v1/acl/identity/client-introduction-token \
    >> intro_token.jwt

Response:

{
  "JWT": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgyNjExLTNkZjItMWM3My1kMDEwLTA1YmUxNmQ3NTBlOCIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJub21hZHByb2plY3QuaW8iLCJleHAiOjE3NjM1MDA5MTgsImlhdCI6MTc2MzUwMDYxOCwianRpIjoiMzBkYjU0OGUtNzQ1OS05MGVlLWYwYmItMjk1MGIzNDJjMTQ2IiwibmJmIjoxNzYzNTAwNjE4LCJub21hZF9ub2RlX25hbWUiOiIiLCJub21hZF9ub2RlX3Bvb2wiOiJkZWZhdWx0Iiwic3ViIjoibm9kZS1pbnRyb2R1Y3Rpb246Z2xvYmFsOmRlZmF1bHQ6OmRlZmF1bHQifQ.acAuFxXshidUF_61l3HFgI0W_eW9ag5J9jptklNb5was2go5E8IHtvueWeYOMcsZakegLX8GNMy8CmBhtSRmmvzWxCoXf5CDHo9jqyxTrFc8yWFozbCucieTWBsyyKMqgkgf52QCm5owO0Kw40AzDK0sqkYrmzh4FrXCruomk0Zmmw9nHfactLRkT4PfYE_RpvMp7ptNkXNlEuEypZ-oVGxOK1maXZ51F0A9xrrMqxBroIfpusuGj8OI4j8VCjDnTKlI1wPWPvNVdirpt3vWp3FdZ-OCTPIGD7wmTzzhdec-cADWrQOxuczmJPoH0U4iF7AdNLswJGhhIKyN4ktVbA"
}

Generate Nomad Client Introduction Token via CLI

In addition to generating the token via API, you can also generate it via the CLI.

Similar to the API, you can pass in additional parameters if needed, but in this case we will just generate a basic token.

nomad node intro create  > intro_token.jwt

The content of the intro_token.jwt file will be similar to this:

 "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgyNjExLTNkZjItMWM3My1kMDEwLTA1YmUxNmQ3NTBlOCIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJub21hZHByb2plY3QuaW8iLCJleHAiOjE3NjM1MDMwNzgsImlhdCI6MTc2MzUwMjc3OCwianRpIjoiZDlhZmRmZjEtMTcxYy1lYWJjLTU4MmEtZjc3NmJjNzY2N2I0IiwibmJmIjoxNzYzNTAyNzc4LCJub21hZF9ub2RlX25hbWUiOiIiLCJub21hZF9ub2RlX3Bvb2wiOiJkZWZhdWx0Iiwic3ViIjoibm9kZS1pbnRyb2R1Y3Rpb246Z2xvYmFsOmRlZmF1bHQ6OmRlZmF1bHQifQ.P9xDbK9dlD8g7XvZ8Wq87mxddCTniaHP0qsR5ovGw9H6KtG6-rHpkYYzKgHgQBVXDUnQjo29yUBLDbZFsa2woub4bX_Qv3LH1TqNq1eAJ3W3W0sHBBj_NZx9GKwW_5MmdlXWe9_tgQVpTM19x1f0Tu8qGRZsQAvZWWhMJZQr9Ad32OP492CT5r_8aZafYCyZfJJqr8puFBFe5_PsEQ5uL-BXT-L8owE41OV9kUZ8zUF3YG75hWCzIcZp-nZHuqzOzqIElXA83V-u4GObrOZKIltkazIbH-JZ20XTA83b8YSh477XQ7OtI1wBD_fKZOXb5ehl7gjFocheSN9LgSdHFg"

With that, we are ready to use this token when starting up our Nomad client agent. In both of these we only want the value of the JWT field and not the surrounding JSON structure.

Using the Client Introduction Token

To use the client introduction token, we have a couple of options.

Set the -client-intro-token flag when starting up the Nomad agent.
Place the token in a file named intro_token.jwt within the client's state directory (default is <data_dir>/<client_state_dir>/).
Set the NOMAD_CLIENT_INTRO_TOKEN environment variable.

For this, I set the token with the environment variable and start up the Nomad client agent again:

nomad agent -config /etc/nomad.d/nomad.hcl
==> Loaded configuration from /etc/nomad.d/nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.39.133:4646
            Bind Addrs: HTTP: [0.0.0.0:4646]
                Client: true
             Log Level: INFO
                Region: global (DC: dc1)
                Server: false
               Version: 1.11.0

==> Nomad agent started! Log data will stream in below:
 ...
    2025-11-19T02:38:56.724Z [INFO]  client: setting node identity token
    2025-11-19T02:38:56.724Z [INFO]  client: started client: node_id=01f47f9a-7281-c450-6db0-3220fdc016a0
    2025-11-19T02:38:56.728Z [INFO]  client: setting node identity token
    2025-11-19T02:38:56.735Z [INFO]  client: node registration complete
    2025-11-19T02:39:03.193Z [INFO]  client: node registration complete

And there we have it! The client has successfully joined the Nomad cluster using the client introduction token.

Conclusion

The client node introduction feature in Nomad 1.11.x adds an important layer of security to your Nomad clusters. By requiring clients to present a valid introduction token, you can better control which nodes are allowed to join your cluster and node pools. This is especially useful in environments where security is a top priority.

GitHub Actions Pipeline

lykins — Tue, 19 Nov 2024 03:11:59 +0000

At this stage, my setup is complete, and I’m ready to start building images. To automate the image factory, I connected two GitHub Actions workflows in a chain. While additional workflows could be introduced for more specialized images, these are general-purpose images with a basic setup, so a simple two-workflow approach suffices.

Workflows

Base Image

The first workflow focuses on creating base images. These images are hardened according to enterprise policies or industry standards, such as NIST, PCI, or other relevant frameworks. In addition to hardening, I ensure that essential infrastructure or application monitoring tools are integrated into the image.

For base images, I worked with the following operating systems:

Red Hat 9.3
Ubuntu 24.04
Windows Server 2022

In terms of platforms, the workflow was configured to build images for AWS, Azure, and vSphere for my presentation.

Once a base image was successfully built, its metadata was sent to their corresponding HCP Packer buckets.

App Image

Here's an updated and refined version of your section:

If the base images are successfully built, the next stage focuses on creating app images. This step layers additional components onto the base image. For Linux images, this might involve setting up Docker or Podman to enable running containerized workloads. For Windows images, configuring IIS could be a suitable solution for hosting web applications.

In my presentation, this is where I concluded the process. However, I did build a Nomad-specific image based on my Docker images to demonstrate the concept.

In a real-world scenario, I would go further by adding additional layers to specialize the image. This approach simplifies the "last mile" provisioning and deployment, ensuring that images are fine-tuned for their specific use cases.

Workflow

Credit

There is an opportunity for an end user to select which image or platform to build on, if looking to do a one-off. Most of the process originated from this repository : packer-images

Inputs

For both base and app workflows, the following is set up:

inputs:
      singleBuild:
        type: choice
        description: image override
        required: false
        options:
          - all
          - rhel
          - ubuntu
          - windows
      platform:
          type: choice
          description: platform override
          required: false
          options:
            - all
            - vsphere
            - aws
            - azure

If triggering a workflow manually, it would look like this:

Build Steps

The first step runs on a GitHub Actions hosted runner and will output which images and platforms to build.

If All images were selected, then RHEL, Ubuntu, and Windows images will be built in AWS, Azure, and vSphere. You can override which image to build or what platform to build on, in any mix.

selectimages:
    runs-on: "ubuntu-latest"
    steps:
      - name: get-images
        id: get-images
        run: |
          if [[ "${{ inputs.singleBuild }}" != "all" ]] && [[ -n "${{ github.event.inputs.singleBuild }}" ]] ; then
            export IMAGES=$(echo ${{ inputs.singleBuild }} | jq -R '["\(.)"]')
            echo "images_out"=$IMAGES""
            echo "images_out"=$IMAGES"" >> $GITHUB_OUTPUT
          else
            export IMAGES=$(echo $DEFAULT_IMAGES | jq -R 'split(", ")')
            echo "images_out"=$IMAGES""
            echo "images_out"=$IMAGES"" >> $GITHUB_OUTPUT
          fi

          if [[ "${{ inputs.platform }}" != "all" ]] && [[ -n "${{ github.event.inputs.platform }}" ]] ; then
            export PLATFORMS=$(echo ${{ inputs.platform }} | jq -R '["\(.)"]')
            echo "platforms_out"=$PLATFORMS""
            echo "platforms_out"=$PLATFORMS"" >> $GITHUB_OUTPUT
          else
            export PLATFORMS=$(echo $DEFAULT_PLATFORMS | jq -R 'split(", ")')
            echo "platforms_out"=$PLATFORMS""
            echo "platforms_out"=$PLATFORMS"" >> $GITHUB_OUTPUT
          fi
    outputs:
      images: ${{ steps.get-images.outputs.images_out }}
      platforms: ${{ steps.get-images.outputs.platforms_out }}

The next step takes outputs from the first step. I use a matrix strategy with image + platform as the combinations.

I had never tried to do this before, but was surprised it worked runs-on: ["${{matrix.platform}}"]. This will ensure my AWS builds will run on my privately networks on AWS, same with Azure and vSphere.

build:
    runs-on: ["${{matrix.platform}}"]
    needs: [selectimages]
    env: 
      HCP_PACKER_BUCKET: ${{matrix.platform}}-${{matrix.image}}
      PLATFORM: ${{matrix.platform}}
    strategy:
      fail-fast: false
      matrix:
        image: ${{fromJson(needs.selectimages.outputs.images)}}
        platform: ${{fromJson(needs.selectimages.outputs.platforms)}}
    steps:
      - uses: FraBle/clean-after-action@v1

      - name: Setup `packer`
        uses: hashicorp/setup-packer@main
        id: setup
        with:
          version: latest

      - name: Checkout code
        uses: actions/checkout@v4.1.1

      - name: Image Builder
        run: |

          packer init -force -var-file=${BUILDPATH}/${LAYER}/${{matrix.image}}/variables/example.pkrvars.hcl ${BUILDPATH}/${LAYER}/${{matrix.image}}/.

          packer build -force -var-file=${BUILDPATH}/${LAYER}/${{matrix.image}}/variables/example.pkrvars.hcl ${BUILDPATH}/${LAYER}/${{matrix.image}}/.

Further Automation

Here's an updated and polished version of your section:

As the process transitions towards production, image building can be triggered on a schedule. While monthly builds might have sufficed in the past, the frequency should ideally increase to weekly. This ensures that security patches and updates are consistently applied. Since the process is automated and runs in the background, this cadence is both practical and efficient.

For testing my images, I’ve utilized the HCP Packer + Terraform integration, which allows me to automatically provision and validate my images. Although I haven’t fully implemented an automated testing pipeline yet, I’ve explored tools like cnspec by Mondoo. This tool integrates with Packer to scan images for compliance during the build process, offering a promising solution for ensuring quality and adherence to security standards.

Multipurpose Packer Templates

lykins — Sat, 16 Nov 2024 04:04:39 +0000

We've now gotten to a point that I have probably the most experience with, and that is with Packer. After Terraform, Packer was the next HashiCorp tool that I picked up. Thankfully, the syntax moved from JSON to HCL a couple of years ago at this point.

What is a Multipurpose Packer Templates

Before I get into the weeds, making hyper-generalized, multipurpose templates might be overkill. Really, my goal here is to write a base template which will work with multiple clouds and on premise hosting solutions. This of it as a way to jumpstart your image building process.

Really what I would like to try to do here is to build just a few templates which can handle all my Packer builds. Think of it as a module in Terraform. A module should be reusable, dynamic, and flexible.

My goal is two templates.

One for base builds.

One for app builds.

Base builds will run from a source cloud image or iso, then register with HCP Packer.

App builds will leverage HCP Packer to pick up the parent image and build upon it.

Additionally, I will only want a single template that will run both Windows and Linux builds. The easiest way to do this would be Ansible, then you can use the

Limitations

The downside with Packer are some of its limitations, specifically with meta-arguments.

As much as I would like to have a count on a provisioner, that is not an option.

Workarounds

So there are a couple of workarounds.

Dynamic

Dynamic blocks are similar to Terraform, where you can run a foreach against it, and it will loop through the input.

Here is an example of a vsphere-iso source block where I loop a map variable to create add a disk to the image I am building. For example, if you want to have a disk just for the os and a disk for application data, you can now loop it in a dynamic block.

variable "vsphere_storage_config" {
  description = "Configuration for vSphere storage."
  type = map(object({
 disk_size = number
 disk_thin_provisioned = bool
  }))
  default = {
    0 = {
disk_size = 50000
disk_thin_provisioned = true
    }
    1 = {
disk_size = 20000
disk_thin_provisioned = true
    }
  }
}

The variable gets injected into here and the result if leaving the default, this would create 2 disks. If you need more or less, you can specify in the respective var-file.

  disk_controller_type = var.vsphere_disk_controller_type
  dynamic "storage" {
  for_each = var.vsphere_storage_config
    content {
      disk_size             = storage.value.disk_size
      disk_thin_provisioned = storage.value.disk_thin_provisioned
    }
  }

Additionally, you can use dynamic where you can as a toggle. For example, if you are or are not using HCP Packer, you can enable it or not.

  dynamic "hcp_packer_registry" {
    for_each = var.hcp_packer_enabled ? [1] : []
    content {
      bucket_name = var.vsphere_bucket_name
      description = var.vsphere_bucket_description
      bucket_labels = merge(var.vsphere_hcp_bucket_labels, {
        "role"       = var.role
        "os"         = var.os
        "os_version" = var.os_version
        "os_type"    = var.os_type
        }
      )
      build_labels = merge(var.vsphere_hcp_build_labels, {
        "packer_version" = packer.version
        }
      )
    }
  }

Dynamic blocks are useful if they can be used, but for provisioners you will need to rely on other solutions.

Only

So for my HashiConf talk, I decided to use Ansible, since it can run on both Windows and Linux, but if using powershell vs shell provisioners in the same build, then it can be tricky.

This is the solution I came up with, and it has seemed to work so far.

  # Probably not the best method to get this to work, but it works for now. 
  provisioner "shell" {
    only              = var.os_type == "linux" ? ["vsphere-iso.this"] : ["foo.this"]
    scripts           = var.build_shell_scripts
    environment_vars  = var.build_shell_script_environment_vars
    valid_exit_codes  = var.build_shell_script_exit_codes
    execute_command   = var.build_shell_script_execute_command
    expect_disconnect = var.build_shell_script_expect_disconnect
  }

  provisioner "powershell" {
    only              = var.os_type == "windows" ? ["vsphere.this"] : ["foo.this"]
    scripts           = var.build_powershell_scripts
    environment_vars  = var.build_powershell_script_environment_vars
    use_pwsh          = var.build_powershell_script_use_pwsh
    valid_exit_codes  = var.build_powershell_script_exit_codes
    execute_command   = var.build_powershell_script_execute_command
    elevated_user     = build.User
    elevated_password = build.Password
    execution_policy  = var.build_powershell_script_execution_policy
  }

Additional Support

There are a couple of other ways I keep my templates generalized.

Really the biggest thing is the structure and layout of my repository I build my images from.

I have a Packer directory which has three subdirectories.

builds - where my packer templates live - base and app.
pkrvars - where my var-files live which make the templates plug and playable.
shared - shared are anything which are can be used across multiple packer builds, no matter the source (aws, azure, gcp, vsphere) or the os type and distribution(linux or windows, rhel or debian).

The tree below is a work in progress, which I will later reference, but it is close to the final product.

└── packer
    ├── builds
    │   ├── app
    │   │   ├── build.pkr.hcl
    │   │   ├── hcp.pkr.hcl
    │   │   ├── locals.pkr.hcl
    │   │   ├── plugins.pkr.hcl
    │   │   ├── source_aws.pkr.hcl
    │   │   ├── source_azure.pkr.hcl
    │   │   ├── source_gce.pkr.hcl
    │   │   ├── variables_aws.pkr.hcl
    │   │   ├── variables_azure.pkr.hcl
    │   │   ├── variables_common.pkr.hcl
    │   │   └── variables_hcp.pkr.hcl
    │   └── base
    │       ├── build.pkr.hcl
    │       ├── locals.pkr.hcl
    │       ├── plugins.pkr.hcl
    │       ├── source_aws.pkr.hcl
    │       ├── source_azure.pkr.hcl
    │       ├── source_gce.pkr.hcl
    │       ├── variables_aws.pkr.hcl
    │       ├── variables_azure.pkr.hcl
    │       ├── variables_common.pkr.hcl
    │       └── variables_hcp.pkr.hcl
    ├── pkrvars
    │   ├── os
    │   │   ├── linux
    │   │   │   ├── debian
    │   │   │   │   ├── base.pkrvars.hcl
    │   │   │   │   ├── nomad.pkrvars.hcl
    │   │   │   │   ├── packer.pkrvars.hcl
    │   │   │   │   ├── terraform.pkrvars.hcl
    │   │   │   │   └── vault.pkrvars.hcl
    │   │   │   └── rhel
    │   │   └── windows
    │   │       └── base.pkrvars.hcl
    │   └── sources
    │       └── sources.pkrvars.hcl
    └── shared
        ├── README.MD
        ├── bootstrap
        │   └── bootstrap_win.txt
        ├── files
        ├── playbooks
        │   └── readme.md
        └── scripts
            ├── debian
            │   ├── base.sh
            │   ├── deprovision-aws.sh
            │   ├── deprovision-azure.sh
            │   ├── deprovision-google.sh
            │   ├── docker.sh
            │   └── nomad.sh
            ├── rhel
            └── windows
                ├── base.ps1
                ├── deprovision-aws.ps1
                ├── deprovision-azure.ps1
                └── deprovision-google.ps1

Shared Resources

When using shared scripts or files, I would recommend using environment_vars on the provisioners and have them specific for the build or source platform you will build on.

For files, use the templatefile function to make a more dynamic file based on the image. So instead of multiple userdata files or autounattend files, you use the templatefile function to customize it.

I am going to call it a night here. My little one is teething and deciding to keep us up at night, so I might try to get to bed.

My newest Packer project is here: plug-and-play-packer.

In this I have all three major clouds - AWS, Azure, GCP. I will also try to get vSphere in, as well.

But more on that later, wife will get on me for being up late soon.

cheers, lykins.

Containerizing GitHub Runner

lykins — Sun, 10 Nov 2024 04:28:56 +0000

So containerizing a custom runner image actually came out a bit simpler than I had expected, this is mostly since others did the heavy lifting for me :).

Credit

All credit goes to the following posts:

How to containerize a GitHub Actions self-hosted runner

Deploying Self-Hosted GitHub Actions Runners with Docker

Changes

So I did make a few changes when I was building my runners.

Built with Packer

This gave me an opportunity to play with the Packer Docker Plugin.

Customized

Since I will be using Ansible Playbooks during my Packer builds, I added it into the image versus setting it up every time the pipeline runs.

In addition to Ansible Playbooks, I add/update the following:

Runners are more customizable and are ephemeral.

Labels: allow custom labels to be added when registering the runner.
Ephemeral: runner will exit after it fails or completes a run.
- Nomad will bring a runner back up to replace.
- Runner would deregister itself from GitHub.

Configuration

./config.sh --url https://github.com/${REPOSITORY} \
    --name ${RUNNER_NAME} \
    --labels ${RUNNER_LABELS} \
    --token ${REG_TOKEN} \
    --unattended \
    --ephemeral

Cleanup

cleanup() {
    echo "Removing runner..."
    ./config.sh remove \
    --unattended \
    --token ${REG_TOKEN}
}

trap 'cleanup' EXIT

Building the Runner

If you are not familiar with Dockerfiles, then you should be fine. I found this to be straightforward whether you have experience or if you are a beginner. I recommend checking out the Packer Docker plugin documentation.

All the code and configuration can be found here:

Runner Image

Really the only major thing you'll need outside your GitHub account is a registry to drop it in. I used GitHub as well in this case. You can also build locally.

Step 1

Add your plugins. I only used Docker, so that's it there.

packer {
  required_plugins {
    docker = {
      source  = "github.com/hashicorp/docker"
      version = "~> 1"
    }
  }
}

Step 2

Like other Packer builds, configure your source block.
Keeping it like GitHub Actions and sticking with Ubuntu. Most other distributions I found to work.

I also specify the USER and script to run when it starts. The script will register the runner once the container is up.

source "docker" "actions-runner-builder" {
  image  = "ubuntu:latest"
  commit = true
  changes = [
    "USER runner",
    "ENTRYPOINT [\"/start.sh\"]",
    "LABEL runner_version=${var.runner_version}",
  ]
}

Step 3

This all goes pretty quickly. The next step is the build block.

First provisioner is to drop the start.sh file onto the image.

  provisioner "file" {
    source      = "packer/builds/github-actions-images/builder/scripts/start.sh"
    destination = "/start.sh"
  }

The next provisioner will configure the runner.

  provisioner "shell" {
    environment_vars = [
      "RUNNER_VERSION=${var.runner_version}",
      "DEBIAN_FRONTEND=noninteractive",
    ]
    script = "packer/builds/github-actions-images/builder/scripts/install.sh"
  }

For my custom image, I wanted to point out three packages.

xorriso - for creating iso images (vSphere intended).
unzip - used when setting up Packer in the workflows.
git - used in packer builds to tag images and metadata.
ansible - most of the builds I ran I used ansible playbooks.

The script itself...

#!/bin/bash -e

# This script sets up the environment for installing GitHub Actions runner.
#
# Environment Variables:
#   DEBIAN_FRONTEND - Set to 'noninteractive' to avoid interactive prompts during package installation.
#   RUNNER_ARCH - Determines the architecture of the system using dpkg.
#   RUNNER_VERSION - Specifies the version of the GitHub Actions runner to install. Defaults to 2.319.1 if not set.
export DEBIAN_FRONTEND=noninteractive
RUNNER_ARCH=$(dpkg --print-architecture)
RUNNER_VERSION=${RUNNER_VERSION:-2.319.1}

# This script checks if the environment variable RUNNER_ARCH is set to "amd64".
# If it is, the script changes the value of RUNNER_ARCH to "x64".
# Finally, it prints the value of RUNNER_ARCH to the console.
if [ "$RUNNER_ARCH" = "amd64" ]; then
  RUNNER_ARCH="x64"
fi
echo "RUNNER_ARCH: $RUNNER_ARCH"

# This script determines the operating system of the runner by using the `uname -s` command,
# converts the output to lowercase, and stores it in the RUNNER_OS variable.
# It then prints the determined operating system.
RUNNER_OS=$(uname -s | tr '[:upper:]' '[:lower:]')
echo "RUNNER_OS: $RUNNER_OS"

# This script creates a new user named 'runner' with a home directory.
useradd -m runner

# This script updates the package lists, upgrades installed packages,
# and installs a set of essential development tools and libraries.
apt-get update -y && apt-get upgrade -y &&
  apt-get install -y --no-install-recommends \
    curl \
    jq \
    build-essential \
    libssl-dev \
    libffi-dev \
    python3 \
    python3-venv \
    python3-dev \
    python3-pip \
    unzip \
    libicu-dev \
    nodejs \
    xorriso \
    git

# This installs Ansible.
apt-get update -y
apt-get install software-properties-common -y
add-apt-repository --yes --update ppa:ansible/ansible
apt-get install ansible -y

# This script installs the GitHub Actions runner.
# It performs the following steps:
# 1. Changes the directory to /home/runner.
# 2. Creates a new directory named actions-runner.
# 3. Changes the directory to actions-runner.
# 4. Downloads the specified version of the GitHub Actions runner tarball from GitHub.
# 5. Extracts the contents of the downloaded tarball.
# 6. Removes the downloaded tarball to clean up.
#
# Environment variables required:
# - RUNNER_VERSION: The version of the GitHub Actions runner to install.
# - RUNNER_OS: The operating system of the runner (e.g., linux, osx).
# - RUNNER_ARCH: The architecture of the runner (e.g., x64, arm64).
cd /home/runner && mkdir actions-runner && cd actions-runner &&
  curl -O -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-${RUNNER_OS}-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz &&
  tar xzf ./actions-runner-${RUNNER_OS}-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz &&
  rm -rf actions-runner-${RUNNER_OS}-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz

# This script changes the ownership of the home directory of the 'runner' user,
# installs dependencies required for GitHub Actions runner using the provided script,
# and then cleans up the APT cache to free up space.
chown -R runner ~runner &&
  /home/runner/actions-runner/bin/installdependencies.sh &&
  rm -rf /var/lib/apt/lists/*

# This script changes the current directory to the root directory
# and then grants execute permissions to the start.sh script located in the root directory.
cd / &&
  chmod +x ./start.sh

Last step in the build is the tag and push the image.

  post-processors {
    post-processor "docker-tag" {
      repository = "${var.registry_server}/${var.registry_username}/actions-runner-builder"
      tags       = concat(["latest"], var.additional_tags)
    }
    post-processor "docker-push" {
      login          = true
      login_username = var.registry_username
      login_password = var.registry_token
      login_server   = var.registry_server
    }
  }

That is it :).

Running the Runner

So a couple of variables need passed when running the container. Only two are required - REPO and TOKEN.

REPO will specify the repository to add the runner to.
TOKEN is your GitHub Personal Access Token.

Additionally, NAME and LABELS can be added to better identify your runner or use in your Actions Workflows later on.

Once you run it you should see it register and connect...

If you check in your repository you will now see the registered runner.

Stopping the Runner

Since the runners are ephemeral, once a workflow run is done in GitHub Actions, it would automatically exit and the process would deregister it.

If running on Nomad or with Docker Compose, a new runner will spin up. In my case, I will stop it manually and since it is not orchestrated a new one should not spin up.

Well... Guess I am using an unnecessary argument. Nevertheless, it triggers the runner to clean up after itself. Something I am trying to hope my three-year-old would do.

If you check back your repository, the runner should now be removed cleanly. Not garbage or hung registrations left.

That is it. Simple.

Intermission: HashiConf

lykins — Mon, 21 Oct 2024 20:47:27 +0000

Just getting back from HashiConf. After 5ish years of watching it virtually, I was finally able to attend in person. This is also the first time I have also publicly presented, definitely will do it again if I am given the opportunity.

This is me.

Yes, I was wearing a fanny pack. I used it to carry my two of my props on stage. Raspberry Pi and one of my Minicomputers to show what I ran Nomad on.

A big thank you to all who attended my talk in person. Hope it gave some interest to those to pick up and use Nomad.

Setting up the Nomad Cluster

lykins — Sun, 13 Oct 2024 03:25:36 +0000

My first task in setting up my Nomad Image Factory is creating the Nomad cluster.

Lab:

3 : Mini Computers
- N100 CPU
- 16 GB Memory
- 2 : 1GB Nics
Proxmox

Nomad Cluster:

Simple Cluster
- 3 Nodes with Server and Client Role
- 2 CPU
- 4 GB Memory
- Ubuntu 24

Installing Nomad:

I take the easy way to install Nomad.

sudo apt-get update && \
  sudo apt-get install wget gpg coreutils

wget -O- https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
| sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt-get update && sudo apt-get install nomad

After installing you should be able to check the Nomad version to confirm.

Additional Notes:

If you want to do a manual install and setup, I would recommend following this guide : HERE :).

If sticking with using the package manager, just a few notes - atleast on he Ubuntu side. I would not think there is a difference with RHEL-based.

Service

It is setup up with a default config. If this was a server only node, I would have set up a nomad user and group, and change root in both places nomad.

Other note is the nomad environment and config file. They are set up in /etc/nomad.d. If changing you'll need to update those values in the service config.

Config

My favorite thing about Nomad and what drew me to it was its use of HCL. My background prior to messing with Nomad is with Terraform + Packer, so it is a common syntax I am already familiar with.

The common config location is /etc/nomad.d.

For now, I will not change anything in here. I probably will play with TLS down the road, but since this is a lab, I am sticking to the defaults.

Starting up Nomad:

Last item I had to hit was install Docker. Went ahead and did that since I will be running containers on this.

At this point I am ready to get it up and running...

Just to note, I entered in the IP address to bootstrap the servers. Everything else is pretty much the default...

data_dir  = "/opt/nomad/data"
bind_addr = "0.0.0.0"

server {
  enabled          = true
  bootstrap_expect = 1
        server_join {
        retry_join = ["192.168.39.109"]
        }
}

client {
  enabled = true
  servers = ["127.0.0.1"]

Once I got the service running on each I checked the UI to confirm all have been joined...

I did notice when checking the clients and their driver status that I must have missed the Docker engine on one of them. I'll jump back in and get that cleaned up.

Well that is it for the night. It is getting late and HashiConf starts in two days :)

Cheers,
lykins

Intro - Nomad Image Factory

lykins — Wed, 09 Oct 2024 02:43:22 +0000

Howdy.

Just a quick introduction in preparation for my series to expand on my HashiConf talk.

Unfortunately, I had to cut out portions of building a full end to end Nomad Image Factory from my talk, but I am going to use my blog to fill in what was missed.

My series of posts are going to be as follows:

Setting up Nomad - Development Server and Cluster

Preparing for my talk, I did not get the chance to play with Consul, so I am going to give it a shot, as well.

My presentation covers running Nomad with the -dev flag, as well as, bootstrapping a cluster.

For my blog, I am going to use my own homelab to show how lightweight Nomad is.

Homelab Features:

1 - Simple Switch
3 - Simple minicomputers
1 - 13-year-old Lenovo laptop
1 - A former Gaming PC

I think my wife is ready for me to move it somewhere else.

Containerizing GitHub Runner

To be transparent, most of the heavy lifting was based on two blog posts. I updated mine to be ephemeral, so after each successful or failed action, the container will exit out, forcing Nomad to recreate. Also added in a few additional packages which are more custom for my Packer templates.

How to containerize a GitHub Actions self-hosted runner

Deploying Self-Hosted GitHub Actions Runners with Docker

In addition to using ephemeral runners, I used the Docker plugin with Packer to build it. This gave me a chance to finally play with Docker + Packer to build an image. Pretty simple process overall.

Multipurpose Packer Templates

Took a different approach with Packer than I had done in the past.

variable "platform" {
  type        = string
  description = "The list of sources to build."
  default     = env("PLATFORM")
}

locals {
  build_source = lookup(local.sources, var.platform, ["source.amazon-ebs.docker"])
  sources = {
    aws     = ["source.amazon-ebs.docker"]
    azure   = ["source.azure-arm.docker"]
    vsphere = ["source.vsphere-clone.docker"]
  }
}

build {
  sources = local.build_source

This is driven by the choices made by inputs when manually kicking off the pipeline. It also allows scaling Packer templates to be much simpler when adding additional sources into the same template. I do not know why I never tried this approach in the past, but it worked out really well for the demo.

Just a FYI, one of the better repositories for vSphere builds are managed by a few folks from VMware (Broadcom). Really appreciate the work they put in it, a bit more than what the average user would need.

vmware-samples/packer-examples-for-vsphere

GitHub Actions Pipeline

Nothing too major here, in my experience most of my pipeline I try to keep pretty straightforward for simplicity.

The GitHub Actions inputs were inspired by this repo from a HashiCorp team.

tfo-apj-demos/packer-images

Running Jobs on Nomad

I'm going to go over a few examples of how you can run a job. I will deploy my GitHub Actions Runner to a cluster with the following methods:

UI
CLI
Terraform
Nomad Pack

In my talk I believe I covered the UI, CLI and Nomad Pack methods. I might have cut out the CLI for time, but I cannot remember.

Scaling Nomad

I could be wrong, but I think the most common method for scaling with Nomad would be with the Nomad Autoscaler running as a job on the cluster:

hashicorp/nomad-autoscaler

Nevertheless, there are a few options to scale Nomad:

Nomad Autoscaler Overview

Also worth looking at the scaling block since placement has different context on how it scales:

Scaling Block

Welp, that is it for now. It is late, and I am going to call it a night. Hope someone finds this useful, but my wife also thinks I need to improve my writing and communication - which is the main reason I decided to blog.

Take care,
lykins

Terraform Professional + HashiConf Talk

lykins — Wed, 28 Aug 2024 17:02:58 +0000

Just a short little update today. I am going to be documenting a new series of blog posts.

HashiCorp Certified: Terraform Authoring and Operations Professional

Took the test as part of the beta program a while back and finally got a result :)

It was a fun opportunity and would recommend others to give it a shot!

HashiConf Presentation

So back when a call for presenters went out for HashiConf, I put out a short topic on learning Nomad. It is one of the Hashi products which has intrigued me the most. I just have not had a reason or opportunity to work with it. Especially with 2 kids aged 3 and under, the amount of free time I have to learn something new is limited.

Well, unexpectedly for me this happened and now I am committed to learning a little bit about Nomad:

So if you happen to be in Boston for HashiConf, come stop by and say hi!

Deploying NeuVector via Helm on minikube

lykins — Thu, 29 Feb 2024 19:26:54 +0000

Summary

The following will walk through the necessary steps to deploy NueVector via Helm. This can be done locally or on a virtual machine. I am using minikube to test on, but K3S/MicroK8s or any other distros will work. Since this is going to be scaled down, we will also limit the replicas.

I'm going to leverage multipass during this to spin up the necessary resource, but any other solution should work.

Prerequisites

Required:

A Virtual Machine
- I will use multipass, which can launch an instance with minikube already installed.

That is really about it for this to get started.

Set up Virtual Machine

Since I have multipass installed, I will launch a new vm using the existing minikube image.

multipass launch -c 8 -m 16G -n demo minikube

multipass launch -c 8 -m 16G -n demo minikube                                                
Waiting for initialization to complete \

Once completed, you should get a launched.

multipass launch -c 8 -m 16G -n demo minikube                                    
Launched: demo

Running a multipass list, will output all the launched virtual machines.

demo                    Running           192.168.64.20    Ubuntu 22.04 LTS
                                          172.17.0.1
                                          192.168.49.1

NeuVector Setup

Connect to the virtual machine, in my case, it is multipass shell minikube.

Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-92-generic aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

  System information as of Thu Feb 29 09:16:40 EST 2024

  System load:                      1.5546875
  Usage of /:                       13.2% of 38.59GB
  Memory usage:                     6%
  Swap usage:                       0%
  Processes:                        199
  Users logged in:                  0
  IPv4 address for br-1746f5f95e03: 192.168.49.1
  IPv4 address for docker0:         172.17.0.1
  IPv4 address for enp0s1:          192.168.64.20
  IPv6 address for enp0s1:          fd3c:28b:5cc5:4064:5054:ff:fe87:5be

NeuVector Setup - minikube

minikube is already started on the new instance; however, I am going to bump up CPUs and Memory for it.

If needing to install minikube, check out the documentation.

First, stop minikube:

ubuntu@demo:~$ minikube stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.

Update CPUs:

ubuntu@demo:~$ minikube config set cpus 4
❗  These changes will take effect upon a minikube delete and then a minikube start

Update Memory:

ubuntu@demo:~$ minikube config set memory 8192
❗  These changes will take effect upon a minikube delete and then a minikube start

Delete exiting minikube:

ubuntu@demo:~$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/ubuntu/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

Start up new minikube:

ubuntu@demo:~$ minikube start
😄  minikube v1.32.0 on Ubuntu 22.04 (arm64)
✨  Automatically selected the docker driver. Other choices: ssh, none
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=4, Memory=8192MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

minikube should be up running, once connected.

You can check with the following command:

minikube status

ubuntu@demo:~$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

If looking to play with minikube more, there are additional add-ons which can be installed, in this case, we will leave the defaults, but metrics-server and dashboard are typical.

NeuVector Setup - kubectl

This image also comes with kubectl setup:

ubuntu@demo:~$ kubectl version
Client Version: v1.28.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

NeuVector Setup - helm

Helm is not installed, but can be quickly set up:

ubuntu@demo:~$ helm version
Command 'helm' not found, but can be installed with:
sudo snap install helm

To install helm on this Ubuntu instance sudo snap install helm --classic.

ubuntu@demo:~$ sudo snap install helm --classic
Download snap "core22" (1125) from channel "stable"

Once install is complete, you can check the version:

ubuntu@demo:~$ helm version
version.BuildInfo{Version:"v3.14.2", GitCommit:"c309b6f0ff63856811846ce18f3bdc93d2b4d54b", GitTreeState:"clean", GoVersion:"go1.21.7"}

NeuVector Setup - Helm Install

Add the helm repo:

helm repo add neuvector https://neuvector.github.io/neuvector-helm/

For this, I'm going to use the latest version, but other older versions and development version can be listed:

helm search repo neuvector --devel -l

This is the latest as of 29 February 2024 -- Leap Day!:

ubuntu@demo:~$ helm search repo neuvector
NAME                CHART VERSION   APP VERSION DESCRIPTION
neuvector/core      2.7.3           5.3.0       Helm chart for NeuVector's core services
neuvector/crd       2.7.3           5.3.0       Helm chart for NeuVector's CRD services
neuvector/monitor   2.7.3           5.3.0       Helm chart for NeuVector monitor services

Helm Install:

For setting up NeuVector, it is simple enough that I will keep most of the default values. I am updating the controller and scanner replicas, if leaving the defaults it will nuke your system since minikube is running a single node. This is fine for local and development environments.

helm upgrade --install neuvector neuvector/core --version 2.7.3 \
--set tag=5.3.0 \
--set controller.replicas=1 \
--set cve.scanner.replicas=1 \
--create-namespace \
--namespace neuvector

The readme for the repository will provide additional configuration options:

NeuVector Helm Chart

When running:

ubuntu@demo:~$ helm upgrade --install neuvector neuvector/core --version 2.7.3 \
--set tag=5.3.0 \
--set controller.replicas=1 \
--set cve.scanner.replicas=1 \
--create-namespace \
--namespace neuvector
Release "neuvector" does not exist. Installing it now.
NAME: neuvector
LAST DEPLOYED: Thu Feb 29 09:34:30 2024
NAMESPACE: neuvector
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Get the NeuVector URL by running these commands:
  NODE_PORT=$(kubectl get --namespace neuvector -o jsonpath="{.spec.ports[0].nodePort}" services neuvector-service-webui)
  NODE_IP=$(kubectl get nodes --namespace neuvector -o jsonpath="{.items[0].status.addresses[0].address}")
  echo https://$NODE_IP:$NODE_PORT

Accessing the NeuVector User Interface

I am going to port-forward this and access it from my local browser. On the virtual machine, run the following command.

kubectl port-forward --address 0.0.0.0 --namespace neuvector service/neuvector-service-webui 8443

ubuntu@demo:~$ kubectl port-forward --address 0.0.0.0 --namespace neuvector service/neuvector-service-webui 8443
Forwarding from 0.0.0.0:8443 -> 8443

This will listen on port 8443 on all addresses (0.0.0.0) and forward to the service : neuvector-service-webui.

Accessing Locally

*On my local browser: *

Note the IP Address I pulled is the virtual machine's private IP address. This can be checked again using multipass list.

Since this is a self-signed certificate, you can ignore the warnings and proceed.

By default, username and password are admin:admin.

Check off on the EULA and you can login.

And voila, update admin password if you plan will continue to use this and you are done.

Additional Steps - Set up mysql container

If looking to test NeuVector a bit more, we will add a MySQL service and run scans on containers and nodes with the NeuVector console.

Add the bitnami repo:

helm repo add bitnami https://charts.bitnami.com/bitnami

Install:

helm install bitnami/mysql --generate-name

In NeuVector Interface

Go to Assets in the navigation pane on the left and select the dropdown. From the dropdown, select containers.

Turn on Auto Scan or perform a manual scan:

Auto Scanning:

Scans will schedule and return back results on completed. Depending on the amount of resources, both scanners and containers, it could take time. Since this is a new cluster, it is relatively quick.

You can filter and view the vulnerabilities which are found:

Go to the Nodes page:

You can see the nodes are also scanned as well for vulnerabilities.

Conclusion

That is about it, a quick and easy way to test out NeuVector. This is really just scratching the surface when it comes to what features and solutions it offers.

Packer Workflows with Jenkins - Building

lykins — Thu, 29 Feb 2024 03:26:47 +0000

This is the second part of a two-parter. Just realized I was about to write a book if I kept going. Now that we have the necessary instances and setup in part 1, we will continue and start building out our images.

Note:
I am using openSUSE Leap, which will soon be discontinued and only maintained for a few years after 15.6. Alternatives would be either Tumbleweed, MicroOS, and Slowroll.

Objectives

Primary:

Create a repository for Packer Templates.
Use Jenkins to build the images.
- I'm going to use a Jenkinsfile as well.

Secondary:

Parallel Build in AWS and Azure.

Prerequisites

Assumption is a basic knowledge of Jenkins, Packer, and Azure. I'm not going to go into much details on how to set up Azure accounts, service accounts, and some other items in this. There are basic examples out the which can be referenced.

Version Control System (GitHub, GitLab, etc.)
Azure Subscription
AWS Account

Repository Setup

*I'm going to keep the repository simple. In most cases the following are needed. *

Folder/Files necessary for CI/CD pipelines
- In this case, I will end up using Jenkinsfile after all. Didn't initially plan to.
- If this was GitHub actions, I'd have a .github with workflows folder. For Jenkins, I'll drop them under a Jenkins directory.
Packer Folder
- In this folder I also have:
  - Builds: for build templates
  - Files: used by the templatefile function or file provisioner.
  - Scripts: for longer scripts which will be more frequently updated. I try to stay away from inline scripts with a provisioner.
  - Var-Files: directory to store var-files which can get pulled in by the pipeline.

*For a tree view: *

.
├── jenkins
│   └── suse-linux-base
└── packer
    ├── builds
    │   ├── builds_suse.pkr.hcl
    │   └── variables.pkr.hcl
    ├── files
    │   └── README.md
    ├── scripts
    │   └── README.md
    └── var-files
        └── README.md

7 directories, 6 files

The repository was made public for anyone that wants to fork it as a demo.

https://github.com/benjamin-lykins/demo-jenkins-packer.git

Jenkins Setup

Create a folder for packer builds.

Jenkins Setup - Credentials

The next item you will want to set up is credentials which will be used by jobs in this folder.

*On the left hand side, select Credentials. *

*Select stores scoped to packer builds or the name of the folder you created. Click on the global domain. *

*Add credentials. *

*For credentials, we will do secret text for all Azure related credentials. Create a secret for each: *
1. Client ID (Service Account ID)
- azure_client_id
2. Client Secret (Service Account Secret)
- azure_client_secret
3. Subscription ID
- azure_subscription_id
4. Tenant ID
- azure_tenant_id

*When all the credentials have been added, it should look like this: *

Jenkins Setup - First Workflow

This first workflow will be simple. Get the latest openSuse image and build and image.

Go into the packer builds folder and create a new pipeline. I called mine suse-linux-base.

*Nothing major setup wise, for Git: *

For Jenkinsfile:

Jenkins - First Build

If forking the repository I provided, the first run should build and run properly.

This is coming from a fresh Azure Account and Subscription.

No resource groups exist.

At this point we can kick off the first build.

In suse-linux-base, select Build Now.

In my case I had a small hiccup, took me a couple of tries to get it working:

Ended up changing the Create Azure Resource step a couple of times, it was running commands out of order, once I got it running, it was doing its thing properly.

packer-rg was created.
packer-rg-temp was created as a temporary resource group to build images in.

Build Completion

You can see the build was completed in the both Jenkins and in the Azure console.

Create VM from Image

From the image, select create VM. Once ready, ssh to connect.

openSUSE Leap 15.5 x86_64 (64-bit)

If you are using extensions consider to enable the auto-update feature
of the extension agent and restarting the service. As root execute:
  - sed -i s/AutoUpdate.Enabled=n/AutoUpdate.Enabled=y/ /etc/waagent.conf
  - rcwaagent restart

As "root" use the:
- zypper command for package management
- yast command for configuration management

Have a lot of fun...

Now that is it. Pretty straight-forward setup. If this is all I wanted, then this would be the stopping point. Very basic usage of Packer, but looking to expand upon what is possible.

Parallel Builds

One of my old jobs I was a vSphere administrator. Every month the virtual machine templates would need to be patched and updated in some way. It was a very manual process and we did not use a tool like Packer to automate or store the configuration in code. It was document and click-ops heavy.

Now, I will not be building images in vSphere, but I will like to build the same image in two public clouds, AWS and Azure.

AWS - Setup

Nothing major needed here, you will need two items. Packer will use default VPC and subnet.

AWS Secret and Key
Subscription to openSUSE Leap

Packer - Setup

To set this up, I will need to add in an additional plugin to the configuration.

Create New Branch

I am going to create a new branch called parallel-aws-azure for this use case.

packer {
  required_plugins {
    azure = {
      source  = "github.com/hashicorp/azure"
      version = "~> 2"
    }
    amazon = {
      source  = "github.com/hashicorp/amazon"
      version = "~> 1"
    }
  }
}

Add a new source block for AWS.

Check for ssh_username: Default user names. In this scenario ec2-user or root would have worked.

source "amazon-ebs" "suse" {
  ami_name      = "suse-image-${local.time}"
  instance_type = "t2.micro"
  region        = "us-east-2"
  source_ami_filter {
    filters = {
      name                = "openSUSE-Leap-*-hvm-ssd-x86_64-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = ["679593333241"]
  }
  ssh_username = "ec2-user"
}

And lastly, update the build block with the new source.

build {
  sources = ["source.azure-arm.suse", "source.amazon-ebs.suse"]

  provisioner "shell" {
    inline = ["echo foo"]
  }
}

Jenkins - Setup

I'll keep the same jenkinsfile for this. All I will update is adding credentials for AWS.

    environment {
        AZURE_SUBSCRIPTION_ID = credentials('azure_subscription_id')
        AZURE_TENANT_ID = credentials('azure_tenant_id')
        AZURE_CLIENT_ID = credentials('azure_client_id')
        AZURE_CLIENT_SECRET = credentials('azure_client_secret')
        AWS_ACCESS_KEY_ID= credentials('aws_access_key_id')
        AWS_SECRET_ACCESS_KEY= credentials('aws_secret_access_key')
    }

Also, ensure thepacker build folder has these keys.

Create New Pipeline

I called mine parallel-build.

Point to the new branch:

Keep the same Jenkinsfile path:

Once ready, hit build and see how it goes.

Packer by default will run both builds concurrently.

[1;32mazure-arm.suse: output will be in this color.[0m
[1;36mamazon-ebs.suse: output will be in this color.[0m

When completed, Packer will output the AMI id and Azure managed image:

Packer:

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs.suse: AMIs were created:
us-east-2: ami-09de620f124c9bc00

Azure:

--> azure-arm.suse: Azure.ResourceManagement.VMImage:

OSType: Linux
ManagedImageResourceGroupName: packer-rg
ManagedImageName: suse-image-20240226195516
ManagedImageId: /subscriptions/****/resourceGroups/packer-rg/providers/Microsoft.Compute/images/suse-image-20240226195516
ManagedImageLocation: East US

In AWS console:

In Azure console:

Nothing too drastically different when it comes to parallel builds in multiple public clouds when setting up in Jenkins.

Conclusion

This is a very surface level example of using Packer with Jenkins. Additional use cases would be:

Layering Builds
- Reference newly build images and add components necessary for hosting an application, such as IIS or Tomcat on the host.
Chaining Pipelines
- This can be done similar to layering builds. Have the base image pipeline run first to build a base image, if success, have a subsequent pipeline trigger to build on the image built in the base pipeline.
Automate Provisioning of Images
- When a new image is built, trigger a Terraform Cloud workspace to run a plan and apply to pick up the latest image.
  - In AWS, Autoscaling Groups can trigger a refresh.
    - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group#triggers
Automate Testing of Images
- I honestly do not have many ideas for this. For me if an image is built with Packer and can be provisioned whether in AWS or Azure, and finally an application can be deployed on it, then that is enough for me. I could see security and compliance tools. Probably just do not have the exposure for this.