DEV Community: Alex Freidah

Rolling Nomad, Consul and Vault Upgrades

Alex Freidah — Fri, 02 Oct 2026 20:30:20 +0000

Seventeen hosts in my cluster run Consul, twelve run Nomad, three run Vault. Upgrading any of them follows the same rules. Servers tolerate running ahead of clients, not the reverse. One server goes down at a time, and only while the rest can still coordinate. The host that coordinates hands that off deliberately rather than by going away.

My binaries are installed by Cinc, the free distribution of Chef. Every host runs cinc-client on a timer, converging it to whatever its cookbooks describe.

The version each tool should be running is not in those cookbooks. It lives in a versions data bag, read by a small library:

# The bag is the only place a version is declared. munchbox-hashi-upgrade
# writes it as part of a rolling upgrade; this repository declares no version
# at all, so there is no second copy to drift from and bumping one is an API
# call rather than a pull request.
#
# Missing pins raise. A pin only goes absent through misconfiguration -- the
# bag comes from the same Chef server as the cookbooks, so a node that cannot
# read it never got far enough to converge -- and falling back to a literal
# would install a stale version instead of naming the problem.
def pinned_version(tool)
  item = data_bag_item(VERSIONS_DATA_BAG, tool)
  version = item['version'].to_s
  raise "munchbox_lib: #{VERSIONS_DATA_BAG}/#{tool} has no 'version' field" if version.empty?

  version
end

The consul, nomad and vault cookbooks all call it the same way, and each restarts its service when the version it installed changes:

vault_install 'vault' do
  version    pinned_version(cookbook)
  bin_path   node[cookbook]['install']['bin_path']
  ...
end

So nothing upgrades by pushing a binary at a host. Changing the data bag changes what every host installs at its next converge, which means an upgrade is: stop the scheduled converges so no host moves on its own, change the pin, then converge the hosts deliberately, in the right order, checking the cluster between each one.

This tool is that sequence.

Three commands

plan reads the cluster and writes a run file. It changes nothing: no pin, no timer, no host. Everything that alters anything is a task in the file it produces.

run carries out a run file, recording the outcome of every task back into it. It resumes, so a run that stopped is continued by naming the same file.

status reads a run file and reports it. No cluster, no credentials, no change. A run that stopped overnight gets read before deciding what to do about it, and reading it should not require the ability to act on it.

hashi-upgrade plan consul --to 2.0.4
  -> consul-munchbox-20261001T062815Z.yaml

hashi-upgrade run    consul-munchbox-20261001T062815Z.yaml
hashi-upgrade status consul-munchbox-20261001T062815Z.yaml

plan names the file after the tool, the cluster it read, and the time it read it. The cluster name keeps two clusters planned from the same directory apart, and the timestamp means planning again writes a new file rather than overwriting the record of a run that already happened.

Task order

survey   freeze-converge         stop the scheduled converges fleet-wide
         set-version-pin         move the pin every converge will read

servers  upgrade-member          each non-coordinating host in turn
         hand-off-coordination   move coordination off the last one
         upgrade-member          that host, last

clients  upgrade-member          each host that carries work, in turn

verify   verify-cluster          read the fleet back against the target
         thaw-converge           release the scheduled converges

The timers stop before the pin moves. A scheduled converge landing between the two installs the new version on a host the run has not reached, out of order and unobserved.

The pin moves once, centrally, before any host is touched, so hosts differ only in when they are converged and never in what they converge toward.

The coordinating host goes last, so the cluster spends most of the server stage with settled coordination.

Verification runs before the timers come back. The per-host gates each proved one host returned, which is not the same as the fleet having arrived: a host the survey missed, or one whose converge was a no-op because the pin never reached it, passes every gate and still runs the old binary.

A plan against my Consul datacenter (and yes, all the bare-metal nodes are named after Law & Order characters):

munchbox: 17 hosts, 1 server failure tolerated
upgrading consul to 2.0.4

servers
  goren            2.0.3
  nomad-server-03  2.0.3     coordinating
  stabler          2.0.4     already at 2.0.4
clients
  cabot            2.0.3
  fontana          2.0.3
  mccoy            2.0.3
  rubirosa         2.0.3
  ...

survey
  Stop scheduled converges across the fleet  [prompt]
  Pin consul to 2.0.4  [typed]
servers
  Upgrade goren to 2.0.4  [prompt, point of no return]
  Upgrade stabler to 2.0.4  [prompt]
  Hand coordination off nomad-server-03  [typed]
  Upgrade nomad-server-03 to 2.0.4  [typed]
clients
  Upgrade cabot to 2.0.4  [prompt]
  Upgrade fontana to 2.0.4  [prompt]
  Upgrade mccoy to 2.0.4  [prompt]
  Upgrade rubirosa to 2.0.4  [prompt]
  ...
verify
  Confirm the fleet is running 2.0.4
  Restore scheduled converges across the fleet

Confirmations are recorded in the file rather than decided by the runner when it reaches them, so which boundaries stop for a person can be reviewed in advance. Two strengths: prompt for anything worth pausing on, and typed for moving coordination and restarting the host that had it. Typed asks for the host's name back.

stabler is marked already at 2.0.4 from an earlier failed attempt. Its task still appears and settles as unnecessary when the run reaches it.

What a survey reads

Topology is never declared. Each tool is asked, and the answer is two reads that have to be matched up.

The two lists overlap. goren and stabler each run a Nomad server and a Nomad client, and a host like that is recorded once, as a server, because it carries one binary and one service. Upgrading it twice would restart one raft member twice and spend the cluster's fault tolerance twice for one host.

They are matched on advertise address rather than on name, because the two reads do not agree on name shape.

Gate conditions

A converge exiting zero does not mean the host is back. The gate after each host polls the cluster until the cluster says so:

func (g *Gate) Server(ctx context.Context, name, target string) error {
        what := fmt.Sprintf("%s rejoining the cluster at %s", name, target)
        arrived := fmt.Sprintf("%s is running %s, healthy, and back in the cluster", name, target)

        return g.await(ctx, what, arrived, func(cluster plan.Cluster) error {
                m, err := member(cluster, name)
                if err != nil {
                        return err
                }

                switch {
                case m.Version != target:
                        return fmt.Errorf("%s is running %s, not %s", name, m.Version, target)
                case !m.Healthy:
                        return fmt.Errorf("%s is not healthy", name)
                // Asked only where coordination runs on a quorum of these members. A
                // cluster whose quorum lives in its storage backend has no voters at
                // all, and requiring one would hold every host here until it timed out.
                case cluster.Votes() && !m.Voter:
                        return fmt.Errorf("%s is not a voter", name)
                case !cluster.Healthy:
                        return fmt.Errorf("%s rejoined but the cluster is not healthy", name)
                case cluster.Tolerance < minTolerance:
                        return fmt.Errorf("failure tolerance is %d, want at least %d", cluster.Tolerance, minTolerance)
                }

                return nil
        })
}

The version proves the host restarted, read from the running agent rather than from the pin. A host that was never restarted reports the old version however healthy it looks.

A timeout carries the condition it gave up on rather than only its duration. "Waited ten minutes" gives nothing to act on; "spent them waiting for cinc-server at 2.0.4, and it is running 2.0.3" identifies the host and the check.

Adding Consul

Consul's autopilot answers in the same shape as Nomad's: healthy, failure tolerance, and a server row carrying version, leader, voter and raft ID. The snapshot a run is generated against is built the same way, and everything downstream is shared without modification: the run file, the runner, the step table, the gates, all three commands.

What is new per tool is one client package and a case here:

// For returns the client for a tool.
//
// A tool with no client is refused by name rather than attempted, so it fails
// here with something to read instead of somewhere deeper holding a nil.
func For(tool plan.Tool, opts Options) (Cluster, error) {
        switch tool {
        case plan.Nomad:
                return nomad.New(nomad.Options{Address: opts.Address, Region: opts.Region})
        case plan.Consul:
                return consul.New(consul.Options{Address: opts.Address})
        case plan.Vault:
                return vault.New(vault.Options{Address: opts.Address})
        default:
                return nil, fmt.Errorf("unknown tool %q; expected nomad, consul or vault", tool)
        }
}

Capabilities that not every tool has stay off the shared interface. Nomad is the only one that places work, so a client able to drain a host implements execute.Drainer and is asked for it at the point of use. The --drain flag is dropped at plan time for a tool that schedules nothing, rather than recorded in the file and ignored.

Vault with Consul storage

My Vault cluster keeps its data in Consul rather than in integrated raft storage, which changes what a survey can read.

There is no raft, so sys/storage/raft/autopilot/state does not exist. There is no failure tolerance to read, no stabilisation timestamp, and no voters. Every node is a standby contending for a lock, and vault operator raft list-peers returns No raft cluster configuration found.

The survey reads sys/ha-status, which lists the HA set from the active node's view. Failure tolerance is derived rather than read: the cluster serves while one node is unsealed and one is active, so what it can afford to lose is every serving node but one.

Voting needed a decision. Treating "Vault does not vote" as a per-tool fact would be wrong, because the same Vault on integrated raft storage does vote. Whether coordination runs on a quorum of the cluster's own members describes the deployment, not the tool, so it is read from the members:

// Votes is whether coordination in this cluster runs on a quorum of its own
// members.
//
// Read from the cluster rather than decided by the tool, because it is a
// property of how the cluster is deployed and not of what it is. A Vault
// cluster keeping its data in Consul has no voters; the same Vault with
// integrated raft storage does. Asking the members means neither case has to
// be configured, and a cluster that is migrated between them is read correctly
// without being told.
//
// One host mid-restart has dropped out of its quorum while its peers have not,
// so any voter makes this a voting cluster.
func (c Cluster) Votes() bool {
        for _, m := range c.Members {
                if m.Voter {
                        return true
                }
        }
        return false
}

Migrating Vault to raft storage later requires no change here.

Restarting a Vault node leaves it sealed until the KMS unseals it. sys/ha-status does not report that, so each node is also asked sys/health at its own address, which does. A sealed node then counts as unhealthy, and a gate waiting on one says it is sealed rather than blaming the version.

A configuration server in the test binary

Everything above moves a pin on a Chef server, so the tests have to prove a pin written there can be read back. Stubbing the HTTP calls proves only that the client called the paths its author expected, which is the same author who wrote the assertions.

cinc-server-ng is that server as a Go library, so the tests run against a real one, in process:

// live starts a configuration server and returns a client that authenticates
// to it as the bootstrap admin.
func live(t *testing.T) *Cinc {
        t.Helper()

        srv, err := server.New(server.Options{Orgs: []string{testOrg}})
        if err != nil {
                t.Fatalf("new configuration server: %v", err)
        }
        if err := srv.Start(); err != nil {
                t.Fatalf("start configuration server: %v", err)
        }
        t.Cleanup(func() { srv.Stop(context.Background()) })

        c, err := New(Options{
                ServerURL:  srv.URL() + "/organizations/" + testOrg,
                ClientName: srv.AdminName(),
                KeyPath:    writeTemp(t, "admin.pem", srv.AdminKey()),
        })
        ...
}

Real data bags and real Mixlib signature verification on both halves of the round trip, in 0.08 seconds, with no Docker and no build tag. These are ordinary unit tests:

func TestPinRoundTrip(t *testing.T) {
        c := live(t)

        if err := c.SetPin(t.Context(), "nomad", "2.0.6"); err != nil {
                t.Fatalf("SetPin: %v", err)
        }

        got, err := c.Pin(t.Context(), "nomad")
        ...
}

The signing is the part this earns. The client signs every request the way Chef expects, and a stub would have accepted a signature that no server would.

Container fleets

Each tool also has a Docker Compose environment: three servers and two clients for Nomad and Consul, three servers for Vault. Each starts a release behind the version to plan toward, since a fleet already at the target plans a run whose every task is a no-op.

They are driven by hand, the same way a real fleet is:

TOOL=consul make cluster-up      # start it, behind the version to plan toward
TOOL=consul make cluster-plan    # survey it and write a run file
TOOL=consul make cluster-run     # drive the file; ARGS=--yes to skip prompts
TOOL=consul make cluster-down    # stop it and discard its state

Each is a separate Compose project on its own configuration-server port, so more than one can be up at once. That matters when a change touches shared code: the Vault work altered the gate conditions and the successor choice for every tool, and having the Nomad and Consul fleets still running is what proved it had not broken them.

The hosts carry stand-ins for the two things a container does not have: a systemctl that answers the timer commands, and a cinc-client that reads the pin and installs it. That covers the orchestration around a converge. Whether a cookbook installs Consul correctly belongs to the cookbook, which is tested on its own.

The Vault environment runs two services that are not part of the fleet being upgraded: a Consul for storage, and a second Vault serving a transit key in place of a cloud KMS, so a restarted node returns unsealed without intervention. A Vault cluster without automatic unsealing cannot be rolled unattended at all.

Four faults found in these environments:

The gates' check for an unhealthy cluster could never fail. Nomad's autopilot answers 200 when the cluster is healthy and 429 when it is not, with the same health document in the body either way. The Go client turns any non-2xx into an error and drops the body, so an unhealthy cluster came back to me as a failed read, never as a snapshot saying Healthy: false.

A gate treats a failed read as "not yet" and polls again, so every snapshot a gate ever received had come from a 200, where healthy is true by definition. The !cluster.Healthy condition was evaluated on every poll and was never once true.

The fix is to pull the body back out of the error, but only for that one status. Anything else stays an error, because a cluster that cannot be read is a different problem from one that reads as unwell:

// unhealthyReply recovers the health reply autopilot sends with a 429.
//
// The status is how autopilot says the cluster is not healthy, which is a
// condition a run waits out rather than an error it stops for. Any other
// status, or a body that will not parse, is left as the error it was: a
// cluster that cannot be read is different from one that reads as unwell.
func unhealthyReply(err error) (*api.OperatorHealthReply, bool) {
        var resp api.UnexpectedResponseError
        if !errors.As(err, &resp) || resp.StatusCode() != http.StatusTooManyRequests {
                return nil, false
        }

        var health api.OperatorHealthReply
        if json.Unmarshal([]byte(resp.Body()), &health) != nil {
                return nil, false
        }

        return &health, true
}

The survey calls it on the one error path it has, and fails as before when it returns false.

Consul uses the same 429 convention for an unhealthy datacenter, but its Go client accepts that status and parses the body rather than erroring on it, so the Consul survey needed no equivalent:

// we use 429 status to indicate unhealthiness
_, resp, err := op.c.doRequest(r)
...
err = requireHttpCodes(resp, 200, 429)

The server gate waited on a timestamp that does not move. An agent that goes down and comes back inside one health interval is never seen unhealthy. StableSince therefore still reads from before the restart, and the gate cannot pass a host that has already arrived.

A failed converge was taken as success. Converge returns a non-zero exit as a result rather than an error, since the command reached the host and reported. The caller discarded it. So a cinc-client that exited 1 counted as done. The run moved on to a gate waiting for a host nothing had installed anything on, timed out there, and reported a version mismatch.

A voter was required in a cluster where nothing votes. Two sites. The gate above is one. The other is choosing which host coordination is handed to, which fails later: after two of three servers are upgraded, at the step marked as the point of no return.

Hand coordination off vault-server-1
unwinding: releasing the converge timers
hand-off-coordination: no healthy voter can take coordination from vault-server-1;
  unfit: [vault-server-2 vault-server-3]

Both sites now call Cluster.Votes() first.

Two faults found in production

Gates gave up after three minutes. A converge installs a couple of hundred megabytes, restarts a service and rejoins a cluster. One host took 3m20s. A gate that gives up inside that fails a run that was going to succeed, and leaves the operator to determine whether the host is wrong or slow. The default is ten minutes.

The pin rolled itself back. Two separate decisions combined into a state with no recovery path.

The pin step recorded a compensation that restored whatever version it found. The reasoning was that an unwinding run turns the converge timers back on. Leaving the pin at the new version would then have every host converge to it unattended, which is what freezing the timers before pinning exists to prevent.

Separately, a resumed run does not repeat a task that already succeeded.

During a 17-host Consul run I left a confirmation prompt unanswered. The run failed, unwound, and restored the pin to 2.0.3.

On restart, the pin step was already recorded as succeeded, so it was skipped. Three hosts then converged onto 2.0.3, the version the fleet was being upgraded away from, and sat at gates waiting for a 2.0.4 that was never coming. Recovering meant editing the data bag by hand.

The fix was to stop treating the pin as a reversible side effect. It is the run's intent. Converges are frozen for the whole run, so nothing reads the pin until the run ends and leaving it set costs nothing. Rolling it back costs twice: a fleet half-converged onto the new version and aimed at the old one converges backwards once the timers return, and a resume cannot repair it.

Two tests had encoded the old behaviour.

Run output

Vault, 2.0.4 to 2.1.1, against the container fleet:

Stop scheduled converges across the fleet
Pin vault to 2.1.1
Upgrade vault-server-2 to 2.1.1
  vault-server-2 | [cinc-client] reading the pin from http://cinc-server:8889/organizations/test
  vault-server-2 | [cinc-client] pinned 2.1.1, running 2.0.4
  vault-server-2 | [cinc-client] fetching https://releases.hashicorp.com/vault/2.1.1/vault_2.1.1_linux_amd64.zip (168 MB)
  vault-server-2 | [cinc-client] downloaded in 4s
  vault-server-2 | [cinc-client] installed Vault v2.1.1
  vault-server-2 | [cinc-client] restarting the agent
  vault-server-2 | [cinc-client] converge complete
  waiting for vault-server-2 rejoining the cluster at 2.1.1
    vault-server-2 is running 2.0.4, not 2.1.1
    vault-server-2 is running 2.1.1, healthy, and back in the cluster (4s)
Upgrade vault-server-3 to 2.1.1
  ...
  waiting for vault-server-3 rejoining the cluster at 2.1.1
    vault-server-3 is running 2.0.4, not 2.1.1
    vault-server-3 is running 2.1.1, healthy, and back in the cluster (13s)
Hand coordination off vault-server-1
  waiting for coordination moving off vault-server-1
    vault-server-1 is no longer coordinating (2s)
Upgrade vault-server-1 to 2.1.1
  ...
  waiting for vault-server-1 rejoining the cluster at 2.1.1
    vault-server-1 is running 2.1.1, healthy, and back in the cluster (4s)
Confirm the fleet is running 2.1.1
Restore scheduled converges across the fleet

every host is on vault 2.1.1

Each gate reports the condition not yet met, then the fact that it now is, with how long it took. vault-server-2 is running 2.0.4, not 2.1.1 is the gate observing a host that has not come back yet.

The download size and timing are printed because curl is silent while it works. The first version printed fetching and then nothing for a minute, which on a cold run is indistinguishable from a hang. I interrupted a working run on that basis.

Resuming

Two outcomes do not settle: failed, where the task reported an error, and active, where it started and never reported back, which is what an interrupt leaves behind. Neither is stepped over on resume, because whether the work took effect is what is unknown.

stopped: upgrade-cinc-server is active

check the host, then: run consul-munchbox-20261001T055728Z.yaml --reset upgrade-cinc-server

The run does not decide on its own that a half-done converge is safe to repeat. Naming the task explicitly, after looking at the host, is what distinguishes a retry from a guess.

Production runs

All three tools have been driven against production: Nomad 2.0.5 to 2.0.7 across 12 hosts, Consul 2.0.3 to 2.0.4 across 17, Vault 2.0.4 to 2.1.1 across 3.

The Consul run exercised the most, since it touches five hosts Nomad does not: the Proxmox hypervisors and the Cinc server. It failed immediately on the Cinc server:

freeze-converge-timers: ssh dial 192.168.68.99:22 as root:
  ssh: handshake failed: ssh: non-certificate host key

The tool verifies host keys against a CA and refuses a bare one. That host was presenting a bare key, with its signed certificate sitting unused on disk, because its role was missing the sshd_ca recipe every other node role carries. One line in a role file. Only a run that dialled every host was going to surface it.

A preflight script exists for exactly this, and it missed it. The script shells out to ssh, which was happy to trust a known_hosts entry. It checks reachability, not the credentials a run actually uses. That gap is still open.

The code

The tool is at munchbox-hashi-upgrade. The design document in that repository covers the sequence, what each tool does differently, and the full list of faults found so far.

Three things are still open. The integration tier covers Nomad only, so the handoff and the coordination gate have no automated test against a real election; they are covered by container fleets driven by hand and by the production runs. A cluster-level barrier between hosts is written and tested, but no step calls it yet. Draining a client before restarting it is implemented and off by default, and has never run against a production fleet.

Originally published at alexfreidah.com.

On-Demand CI Runners on Nomad with Temporal

Alex Freidah — Mon, 28 Sep 2026 00:14:15 +0000

Some of my CI can't run on GitHub-hosted runners. Pushing images to my Docker registry, deploying updated Nomad jobs, and applying Terragrunt, which covers everything from cloud instances and DNS to Vault policies and ACLs, all need to reach the cluster, so they have to run inside it. The usual answer is self-hosted runners sitting in Nomad waiting for work, and on a personal account that gets ugly fast. Outside an org, a self-hosted runner can only be registered to one repo. CI for six repos means six runners, each registered by hand with its own token, and all of them tying up a real slice of a homelab cluster to do nothing most of the day.

I was already running Temporal on the cluster for nightly backups, snapshots, and general maintenance, so I got to thinking I could set up a polling workflow to watch my repos and dispatch a parameterized Nomad job with the right registration token, repo, labels, and image plugged in.

The shape

Temporal Schedule (every 30s)
        |
        v
PollAndDispatch
  - load per-repo config from Consul KV
  - list queued self-hosted jobs per repo
  - count runners already pending/running
  - start one HandleRunner child per missing runner
        |
        v
HandleRunner (one per runner)
  - mint a registration token
  - nomad job dispatch ci-runner (token passed as meta)
  - wait for the allocation to finish
  - stop the dispatched job

The runner itself is a Nomad parameterized batch job. Every dispatch is one ephemeral runner: it registers, takes exactly one job, deregisters, and exits. Restarts and rescheduling are off, because a finished runner should stay finished.

Here's the runner job:

# CI Runner — on-demand ephemeral GitHub Actions runner (parameterized)
#
# Each dispatch spawns one ephemeral self-hosted runner that takes a single job,
# then deregisters and exits. Dispatched by the Temporal poller, which mints the
# registration token and passes it as meta.

job "ci-runner" {
  region      = "global"
  datacenters = ["munchbox"]
  type        = "batch"
  node_pool   = "default"

  # Dispatched per CI run; meta carries the target repo + minted token
  parameterized {
    meta_required = ["repo_url", "runner_token"]
    meta_optional = ["labels"]
  }

  # Default labels when a dispatch omits them
  meta {
    labels = "self-hosted"
  }

  group "runner" {
    count = 1

    # amd64-only image; keep it off arm64 nodes (Pi5s) where it can't run
    constraint {
      attribute = "${attr.cpu.arch}"
      value     = "amd64"
    }

    network {
      mode = "host"
    }

    # One-shot: an ephemeral runner runs a single job then exits; never
    # restart or reschedule a finished/failed runner
    restart {
      attempts = 0
      mode     = "fail"
    }

    reschedule {
      attempts  = 0
      unlimited = false
    }

    task "runner" {
      driver = "docker"

      # WI exchanged for a Vault token so the template below can read the
      # scoped Nomad ACL token. Role + policy live in terragrunt vault-config.
      vault {
        role        = "ci-runner"
        change_mode = "noop"
      }

      # The default WI (identity.env) carries no Nomad policy; the scoped
      # NOMAD_TOKEN templated in below overrides it.
      identity {
        env  = true
        file = true
        aud  = ["vault.io"]
      }

      config {
        image = "registry.munchbox.cc/ci-runner:latest"
        # force_pull so a rebuilt :latest (e.g. a tool version bump) is picked up
        # on the next dispatch instead of a stale cached image.
        force_pull         = true
        image_pull_timeout = "10m"
        volumes = [
          # pki_int signs the Nomad server cert; this CA backs NOMAD_CACERT.
          "/opt/nomad/tls/vault-intermediate-ca.pem:/etc/ssl/certs/munchbox-ca.pem:ro",
        ]
      }

      # Scoped Nomad ACL token (submit-job) for `nomad job validate`/`plan`.
      # Minted by terragrunt nomad-acls -> secret/ci-runner-nomad.
      template {
        data        = <<-EOF
        {{ with secret "secret/data/ci-runner-nomad" }}
        NOMAD_TOKEN={{ .Data.data.nomad_token }}
        {{ end }}
        EOF
        destination = "secrets/nomad.env"
        env         = true
      }

      env {
        RUNNER_SCOPE        = "repo"
        REPO_URL            = "${NOMAD_META_repo_url}"
        RUNNER_TOKEN        = "${NOMAD_META_runner_token}"
        LABELS              = "${NOMAD_META_labels}"
        EPHEMERAL           = "true"
        DISABLE_AUTO_UPDATE = "true"
        RUN_AS_ROOT         = "false"
        RUNNER_NAME         = "ci-runner-${NOMAD_ALLOC_ID}"
        RUNNER_WORKDIR      = "/tmp/runner-work"

        # Nomad API for `nomad job validate`/`plan`; NOMAD_TOKEN from template
        NOMAD_ADDR            = "https://192.168.68.61:4646"
        NOMAD_TLS_SERVER_NAME = "server.global.nomad"
        NOMAD_CACERT          = "/etc/ssl/certs/munchbox-ca.pem"
      }

      # Ephemeral one-shot, so a flat reservation has no idle cost (no need
      # for memory_max/oversubscription)
      resources {
        cpu    = 2000
        memory = 2048
      }
    }
  }
}

Talking to Nomad

Everything the scaler does to Nomad goes through a small wrapper around the official Go client, github.com/hashicorp/nomad/api. Dispatching a runner is a single Jobs().Dispatch call that returns the ID of the concrete dispatched job, and reaping one is a Deregister with purge, so finished dispatch jobs don't pile up in Nomad's state. The two calls worth showing are the ones the reconcile and the wait depend on.

Counting the runners already in flight doesn't need any state of its own. Every dispatched runner is a child job named <parent>/dispatch-…, and its meta still carries the repo and labels it was dispatched with, so Nomad already knows everything the reconcile needs:

// ActiveRunnerSlots returns the dispatch identity of every active (pending or
// running) dispatched child of parentJobID. It lists allocations, keeps those
// whose job is a dispatched child still occupying a slot, and reads each child
// job's repo_url + labels meta. A child whose job info can't be fetched (already
// garbage-collected) is skipped -- it no longer occupies a slot. Ephemeral
// runners aren't bound to a specific job, so the caller reconciles by counting
// these against queued jobs rather than tracking a runner per job_id.
func (n *Nomad) ActiveRunnerSlots(ctx context.Context, parentJobID string) ([]RunnerSlot, error) {
    allocs, _, err := n.client.Allocations().List((&api.QueryOptions{}).WithContext(ctx))
    if err != nil {
        return nil, fmt.Errorf("list allocations: %w", err)
    }
    prefix := parentJobID + "/dispatch-"
    active := make(map[string]struct{})
    for _, al := range allocs {
        if !strings.HasPrefix(al.JobID, prefix) {
            continue
        }
        if al.ClientStatus == api.AllocClientStatusPending || al.ClientStatus == api.AllocClientStatusRunning {
            active[al.JobID] = struct{}{}
        }
    }
    slots := make([]RunnerSlot, 0, len(active))
    for jobID := range active {
        job, _, err := n.client.Jobs().Info(jobID, (&api.QueryOptions{}).WithContext(ctx))
        if err != nil {
            continue
        }
        slots = append(slots, RunnerSlot{
            RepoURL: metaString(job.Meta, "repo_url"),
            Labels:  metaString(job.Meta, "labels"),
        })
    }
    return slots, nil
}

Knowing when a runner is done means checking its allocations. A runner that hasn't been scheduled yet has none, and that counts as not done, so a pending runner is never reaped before it runs:

// RunnerTerminal reports whether a dispatched runner job has finished: every
// allocation is in a terminal client status, or the job is already gone. A job
// with no allocations yet (dispatched but not scheduled) is not terminal, so a
// caller polling this waits for the runner to actually run before reaping --
// never reaping one still pending or mid-job.
func (n *Nomad) RunnerTerminal(ctx context.Context, jobID string) (bool, error) {
    allocs, _, err := n.client.Jobs().Allocations(jobID, false, (&api.QueryOptions{}).WithContext(ctx))
    if err != nil {
        if IsJobNotFound(err) {
            return true, nil
        }
        return false, err
    }
    if len(allocs) == 0 {
        return false, nil
    }
    for _, al := range allocs {
        if al.ClientStatus == api.AllocClientStatusPending || al.ClientStatus == api.AllocClientStatusRunning {
            return false, nil
        }
    }
    return true, nil
}

Finding queued jobs

GitHub has no "list queued jobs" endpoint, so the poller has to build that list itself. It walks the repo's workflow runs that are queued or in_progress, because a run with several jobs can be in progress while one of its jobs is still waiting, and keeps the jobs that are queued and ask for self-hosted. The same job can show up under both run states, so it dedupes by job ID at the end:

func listQueuedSelfHostedJobs(ctx context.Context, cli *github.Client, owner, repo string) ([]QueuedJob, error) {
    var all []QueuedJob
    for _, status := range []string{"queued", "in_progress"} {
        runIDs, err := listWorkflowRunIDs(ctx, cli, owner, repo, status)
        if err != nil {
            return nil, err
        }
        for _, runID := range runIDs {
            jobs, err := queuedSelfHostedJobsForRun(ctx, cli, owner, repo, runID)
            if err != nil {
                return nil, err
            }
            all = append(all, jobs...)
        }
    }
    return dedupByID(all), nil
}

// queuedSelfHostedJobsForRun returns runID's jobs that are still queued and ask
// for a self-hosted runner.
func queuedSelfHostedJobsForRun(ctx context.Context, cli *github.Client, owner, repo string, runID int64) ([]QueuedJob, error) {
    opts := &github.ListWorkflowJobsOptions{Filter: "latest", PerPage: 100}
    var jobs []QueuedJob
    for job, err := range cli.Actions.ListWorkflowJobsIter(ctx, owner, repo, runID, opts) {
        if err != nil {
            return nil, fmt.Errorf("list jobs for run %d in %s/%s: %w", runID, owner, repo, err)
        }
        if job.GetStatus() != "queued" || !slices.Contains(job.Labels, selfHostedLabel) {
            continue
        }
        jobs = append(jobs, QueuedJob{
            ID:     job.GetID(),
            RunID:  job.GetRunID(),
            Name:   job.GetName(),
            Labels: job.Labels,
        })
    }
    return jobs, nil
}

Forgejo does have an endpoint for a repo's runner jobs, but it has its own catch. A job reports waiting from the moment it's created until a runner claims it, and for a moment after the claim the status hasn't caught up yet. Counting that job as queued would dispatch a second runner for work that's already being done, so the Forgejo client also checks whether a task has been assigned:

for _, j := range jobs {
    if !strings.EqualFold(j.Status, statusWaiting) || j.TaskID != 0 {
        continue
    }
    out = append(out, git.QueuedJob{
        ID:     j.ID,
        Name:   j.Name,
        Labels: j.RunsOn,
    })
}

Registering each runner to the right repo

Repos in a personal account can't share self-hosted runners the way repos in a GitHub org can. Each repo needs its own runner registration, so a runner has to be registered to whichever repo queued the job. The scaler does that at dispatch time: it mints a registration token for that repo through a GitHub App installed on all my repos, and passes it to the runner as dispatch meta. Nothing is stored and nothing is set up per repo. Adding a repo to CI is installing the App on it and adding one line to the config below.

The App client itself only ever holds a JWT signed with the App's private key. Every call that touches a repo first mints an installation token scoped to that one repo with the one permission the call needs: administration: write to create a registration token, actions: read to look at the queue. From the shared GitHub client:

// installationClient mints an installation token scoped to repo with perms and
// returns a token-authenticated client. This is the one place the per-call token
// dance lives (SetRepoSecret and the runner methods all share it); the App
// client itself only ever holds the JWT.
func (g *GitHub) installationClient(ctx context.Context, repo string, perms *github.InstallationPermissions) (*github.Client, error) {
    tok, _, err := g.app.Apps.CreateInstallationToken(ctx, g.instID, &github.InstallationTokenOptions{
        Repositories: []string{repo},
        Permissions:  perms,
    })
    if err != nil {
        return nil, fmt.Errorf("mint installation token for %s: %w", repo, err)
    }
    opts := []github.ClientOptionsFunc{
        github.WithTransport(shared.OTelTransport("github", nil)),
        github.WithAuthToken(tok.GetToken()),
    }
    if g.baseURL != "" {
        opts = append(opts, github.WithEnterpriseURLs(g.baseURL, g.baseURL))
    }
    cli, err := github.NewClient(opts...)
    if err != nil {
        return nil, fmt.Errorf("github token client: %w", err)
    }
    return cli, nil
}

// CreateRunnerRegistrationToken mints a short-lived registration token a runner
// uses to join owner/repo, returning the token and its expiry. Requires the App
// installation to grant administration:write.
func (g *GitHub) CreateRunnerRegistrationToken(ctx context.Context, owner, repo string) (string, time.Time, error) {
    cli, err := g.installationClient(ctx, repo, &github.InstallationPermissions{Administration: new("write")})
    if err != nil {
        return "", time.Time{}, err
    }
    tok, _, err := cli.Actions.CreateRegistrationToken(ctx, owner, repo)
    if err != nil {
        return "", time.Time{}, fmt.Errorf("create runner registration token for %s/%s: %w", owner, repo, err)
    }
    return tok.GetToken(), tok.GetExpiresAt().Time, nil
}

The scaler calls this inside the dispatch activity and hands the result straight to Nomad as the runner_token meta. Only the dispatched job ID comes back out of the activity, which keeps the token out of Temporal's workflow history (kept for the namespace's retention period and visible in the UI).

Not every repo can use the App. For a repo I'm only a collaborator on, the App can't be installed, so the tokens have to be set up by hand once. I create a low-privilege personal access token for polling, which only needs Actions: read. Registering a runner needs admin on the repo, so that token has to come from the owner. Both go into Vault, where the scaler reads the polling token and the dispatched job reads the admin token and registers itself. The higher-privilege token never goes to the scaler. The same code also polls my Forgejo mirror, which mints registration tokens per repo the same way the App does.

The scaler reads its config from one JSON document at runners/config in Consul KV. Here is a portion of my Consul KV config:

{
  "afreidah/munchbox": {
    "maxConcurrent": 3,
    "mode": "app"
  },
  "afreidah/nomad-temporal-jobs": {
    "maxConcurrent": 3,
    "mode": "app"
  },
  "alex/s3-orchestrator": {
    "forgejoUrl": "http://forgejo.service.consul:30028",
    "maxConcurrent": 1,
    "mode": "forgejo",
    "profiles": [
      {
        "job": "forgejo-build-runner",
        "label": "build",
        "maxConcurrent": 1
      }
    ],
    "vaultPath": "forgejo/scaler"
  },
  "ev-the-dev/moat": {
    "mode": "vault",
    "profiles": [
      {
        "job": "github-runner-moat-vm",
        "label": "vm",
        "maxConcurrent": 1
      },
      {
        "job": "go-ci-runner",
        "label": "go",
        "maxConcurrent": 3
      },
      {
        "job": "github-runner-moat",
        "label": "moat",
        "maxConcurrent": 2
      }
    ],
    "registerVaultPath": "github/moat-runner",
    "vaultPath": "github/moat-poll"
  }
}

Profiles route jobs to different runner pools by their runs-on labels, first match wins. A plain self-hosted job falls through to the default ci-runner. Caps keep a pool from eating the cluster; anything over the cap just waits in GitHub's queue until a slot frees.

Learning how GitHub assigns jobs to self-hosted runners

My first version tied each runner to a specific queued job. It started one workflow per job and used the job's ID as the workflow's ID, so the same job couldn't be handled twice.

That isn't how GitHub works. You can't control which job a runner picks up. When a repo has several jobs queued with the same labels, GitHub hands a new runner any one of them, not necessarily the one it was started for. So the runner started for the first job could end up running the second. The first job was still waiting, but as far as the scaler knew it had already been handled, and the duplicate check stopped it from starting another runner. That job sat in the queue forever.

The fix was to stop thinking in jobs and reconcile by queue depth. Each tick, the poller groups queued jobs by (repo, labels), counts the runners already pending or running for each group from their Nomad dispatch meta, and starts children only for the difference. If a job gets skipped, it's still queued on the next tick, the count comes up short, and another runner is dispatched. There's no dedup state to get wrong.

Here's that working in the scaler's logs (trimmed to the relevant fields). A job was queued on munchbox, a runner was dispatched for it, and on the next tick the job was still queued because the runner hadn't claimed it yet. The runner already counted as active, so nothing new started:

21:01:01.826  Poll complete                queued=1 active=0 started=1
21:01:02.341  Dispatched ephemeral runner  repo=afreidah/munchbox job=ci-runner labels=[self-hosted] minted=true
21:01:31.853  Poll complete                queued=1 active=1 started=0
21:02:02.026  Poll complete                queued=0 active=0 started=0
21:02:07.465  Runner finished              job=ci-runner/dispatch-1789592462-171d8bee
21:02:07.572  Reaped ephemeral runner      job=ci-runner/dispatch-1789592462-171d8bee

The reconcile itself, from PollAndDispatch. A bucket key is the repo plus its sorted labels, so a runner's dispatch meta and a queued job's runs-on list land in the same bucket regardless of label order:

for _, key := range order {
    b := buckets[key]
    active := activeCounts[key]
    result.QueuedJobs += b.queued
    result.ActiveRunners += active

    // range over a negative shortfall iterates zero times -- an over-covered
    // bucket dispatches nothing.
    needed := b.queued - active
    // Clamp to the pool's concurrency ceiling (profile cap, else the repo-wide
    // cap); overflow stays queued on GitHub.
    rc := repoCfgs[b.repo]
    limit := matchProfile(rc.Profiles, b.labels).MaxConcurrent
    if limit == 0 {
        limit = rc.MaxConcurrent
    }
    if limit > 0 && needed > limit-active {
        needed = limit - active
    }
    for range needed {
        if err := startRunnerChild(ctx, b.repo, b.labels, repoCfgs[b.repo], reapAfter, seq); err != nil {
            logger.Warn("Failed to start runner child", "repo", b.repo, "labels", b.labels, "error", err)
        } else {
            result.RunnersStarted++
        }
        seq++
    }
}

Cleaning up

Each HandleRunner child waits for its runner's allocation to finish and then stops the dispatched Nomad job, so dead dispatch jobs don't pile up. The wait has a one-hour ceiling. A runner that wedges or never claims a job is reaped anyway.

Two details matter here. The dispatch activity never retries, because every attempt creates a new runner, and a lost response followed by a retry would start a duplicate. The reap runs on a disconnected context, so it still happens when the workflow is cancelled or times out. The whole child workflow:

func HandleRunner(ctx workflow.Context, spec RunnerSpec) error {
    logger := workflow.GetLogger(ctx)

    // Dispatch must not be retried: it creates a new runner each attempt, so a
    // lost response on retry would spawn a duplicate. The registration token is
    // minted inside the activity, so it never enters workflow history.
    dispatchCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout:    2 * time.Minute,
        ScheduleToCloseTimeout: 2 * time.Minute,
        RetryPolicy:            shared.NoRetry(),
    })
    var dispatchedID string
    err := workflow.ExecuteActivity(dispatchCtx, a.DispatchRunner, activities.DispatchSpec{
        Repo:        spec.Repo,
        Labels:      spec.Labels,
        Job:         spec.Job,
        Image:       spec.Image,
        MintToken:   spec.MintToken,
        VaultSecret: spec.VaultSecret,
        Mode:        spec.Mode,
        ForgejoURL:  spec.ForgejoURL,
        VaultPath:   spec.VaultPath,
    }).Get(dispatchCtx, &dispatchedID)
    if err != nil {
        return fmt.Errorf("dispatch runner for %s: %w", spec.Repo, err)
    }

    reapAfter := spec.ReapAfter
    if reapAfter <= 0 {
        reapAfter = defaultReapAfter
    }
    // Wait until the runner's alloc goes terminal so we reap promptly, with
    // reapAfter as the backstop ceiling (StartToCloseTimeout): a wedged runner
    // times the wait out and we reap anyway. NoRetry -- a timeout means "reap
    // now", not "wait another ceiling". A cancellation (operator terminate) also
    // falls through to the reap below so the dispatched runner is never orphaned.
    waitCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: reapAfter,
        HeartbeatTimeout:    time.Minute,
        RetryPolicy:         shared.NoRetry(),
    })
    if err := workflow.ExecuteActivity(waitCtx, a.WaitRunnerDone, dispatchedID).Get(waitCtx, nil); err != nil {
        logger.Info("Runner wait ended early (backstop deadline or cancellation); reaping now", "job", dispatchedID, "error", err)
    }

    // Reap on a disconnected context so a closing/cancelled workflow can still
    // stop the Nomad job. The reaper treats an already-gone job as success.
    reapCtx, cancel := workflow.NewDisconnectedContext(ctx)
    defer cancel()
    reapCtx = workflow.WithActivityOptions(reapCtx, shared.QuickActivityOptions())
    if err := workflow.ExecuteActivity(reapCtx, a.ReapRunner, dispatchedID).Get(reapCtx, nil); err != nil {
        return fmt.Errorf("reap runner %s: %w", dispatchedID, err)
    }
    return nil
}

What a run looks like

Here's a real tick that found a build queued on the Forgejo mirror of s3-orchestrator. The output is from a small script that reads the workflow's event history with temporal workflow show and prints each activity's input and result:

PollAndDispatch  ci-runner-scaler-scheduled-2026-09-25T21:01:00Z
One scheduled tick of the CI runner scaler. It reads the repo list, asks each repo's forge for queued self-hosted jobs, counts the runners already running in Nomad, and starts a runner for every job that isn't covered.
started 2026-09-25T21:01:00Z, took 1.6s

[1] LoadConfig
    Read the per-repo runner config from Consul KV.
    10 repos configured:
      afreidah/munchbox               app      cap 3
      afreidah/nomad-temporal-jobs    app      cap 3
      alex/cloudflare-log-collector   forgejo  cap 1  labels: build
      alex/flight-fetcher             forgejo  cap 1  labels: build
      alex/g3                         forgejo  cap 1  labels: build
      alex/munchbox                   forgejo  cap 2  labels: build, ops, cinc, self-hosted, ubuntu-latest
      alex/nomad-temporal-jobs        forgejo  cap 1  labels: build
      alex/oracle-watchdog            forgejo  cap 1  labels: build
      alex/s3-orchestrator            forgejo  cap 1  labels: build
      ev-the-dev/moat                 vault           labels: vm, go, moat

[2] ListQueuedJobs  x10
    Ask each repo's forge for jobs waiting on a self-hosted runner.
      afreidah/munchbox               app      nothing queued
      afreidah/nomad-temporal-jobs    app      nothing queued
      alex/cloudflare-log-collector   forgejo  nothing queued
      alex/flight-fetcher             forgejo  nothing queued
      alex/g3                         forgejo  nothing queued
      alex/munchbox                   forgejo  nothing queued
      alex/nomad-temporal-jobs        forgejo  nothing queued
      alex/oracle-watchdog            forgejo  nothing queued
      alex/s3-orchestrator            forgejo  1 queued
        - Build and push images  [build]
      ev-the-dev/moat                 vault    nothing queued

[3] CountActiveRunners
    Count runners already pending or running in Nomad, per repo and label set.
    Nomad jobs checked: forgejo-build-runner, forgejo-ci-runner, github-runner-moat-vm, go-ci-runner, github-runner-moat
    no runners active

Runners started: 1
    runner-01a0da5f-1719-759d-8044-6ee6aad20d61-0       alex/s3-orchestrator  [build]
    run temporal-steps <id> to see one runner's dispatch, wait, and reap

Result
    repos scanned: 10   queued jobs: 1   active runners: 0   runners started: 1

And the runner it started, from dispatch to cleanup:

HandleRunner  runner-01a0da5f-1719-759d-8044-6ee6aad20d61-0
One ephemeral runner. It dispatches the runner job in Nomad, waits for the runner to finish its CI job, then stops and purges the job.
started 2026-09-25T21:01:01Z, took 125.5s

[1] DispatchRunner
    Mint a registration token (or pass a Vault path) and dispatch one runner job in Nomad.
    repo:    alex/s3-orchestrator
    labels:  build
    job:     forgejo-build-runner
    token:   minted for this runner
    dispatched as forgejo-build-runner/dispatch-1790370061-44e4b15b

[2] WaitRunnerDone
    Wait for the runner's allocation to finish, up to the one-hour ceiling.
    runner: forgejo-build-runner/dispatch-1790370061-44e4b15b
    finished

[3] ReapRunner
    Stop and purge the finished runner job in Nomad.
    stopped and purged forgejo-build-runner/dispatch-1790370061-44e4b15b

Result
    completed

The same run from the scaler worker's logs in Nomad, trimmed to the relevant fields. The runner itself leaves no logs behind, because the reap purges its job and Nomad drops the allocation with it:

21:01:00.156  Polling for queued runners
21:01:01.705  Poll complete                queued=1 active=0 started=1
21:01:01.815  Dispatched ephemeral runner  repo=alex/s3-orchestrator job=forgejo-build-runner labels=[build] minted=true
                                           dispatched=forgejo-build-runner/dispatch-1790370061-44e4b15b
21:01:31.543  Poll complete                queued=0 active=0 started=0
21:02:01.466  Poll complete                queued=0 active=0 started=0
21:02:31.421  Poll complete                queued=0 active=0 started=0
21:03:01.586  Poll complete                queued=0 active=0 started=0
21:03:06.986  Runner finished              job=forgejo-build-runner/dispatch-1790370061-44e4b15b
21:03:07.086  Reaped ephemeral runner      job=forgejo-build-runner/dispatch-1790370061-44e4b15b

The Temporal web UI shows the same thing graphically. These two screenshots are from an earlier build on the same repo, at 20:31.

First the tick. The top of the page is the workflow's input (the scan concurrency) and its result, which says it scanned 10 repos, found 1 queued job, and started 1 runner. The timeline under it has one bar per activity. CountActiveRunners is selected, so the panel at the bottom shows how Temporal ran that step: its timeouts, its retry policy (up to 3 attempts, backing off from 1 second), and its input, which is the list of Nomad runner jobs it checked. The HandleRunner marker at the right edge of the timeline is the child workflow the tick started for the queued build.

Then that child. Its input is the spec the tick handed it: the repo, the build label, which Nomad job to dispatch, and that it should mint a token against the Forgejo instance. The timeline shows the whole life of the runner. DispatchRunner is the short bar at the very start, WaitRunnerDone runs for the two minutes the build took, and ReapRunner is the short bar at the end once the runner exited.

Why Temporal

Mostly because it was already there. Temporal was running my backups and maintenance, and nomad-temporal-jobs already had shared client libraries for Nomad, Consul, Vault, and GitHub. The scaler needed exactly those four, so it was a natural extension: a new worker built out of clients I was already using. The scaler's entire main() is wiring those clients together and registering two workflows:

func main() {
    err := shared.RunWorker(context.Background(), shared.WorkerSpec{
        Service:   "ci-runner-scaler",
        TaskQueue: "ci-runner-scaler-task-queue",
        Register: func(ctx context.Context, slogger *slog.Logger, w worker.Worker) (func(), error) {
            // Vault (Workload Identity); the GitHub App key is pulled through it,
            // so the Nomad job carries only its identity.
            vc, err := vault.NewVaultWithRefresher(ctx, slogger)
            if err != nil {
                return nil, err
            }
            // Reuses the token-renewer App (also needs Administration + Actions perms).
            appPath := cmp.Or(os.Getenv("GITHUB_APP_VAULT_PATH"), "github/token-renewer-app")
            gh, err := git.NewGitHubFromVault(ctx, vc, appPath)
            if err != nil {
                return nil, err
            }
            // Consul KV (per-repo runner config) uses the local agent's default
            // ACL token over host networking -- no per-worker Consul token.
            kv, err := consul.NewConsul(ctx, nil)
            if err != nil {
                return nil, err
            }
            nm, err := nomad.NewNomad()
            if err != nil {
                return nil, err
            }

            // Vault doubles as the vault-mode PAT source: repos the App can't reach
            // carry a runner token in the secret store, read per poll via ReadKV.
            // The default NewPATLister (git.NewGitHubPAT) is left in place.
            acts := activities.New(activities.Config{
                GitHub:      gh,
                KV:          kv,
                Nomad:       nm,
                Vault:       vc,
                ConfigKey:   os.Getenv("RUNNERS_CONFIG_KEY"),
                RunnerJobID: os.Getenv("RUNNER_JOB_ID"),
            })
            w.RegisterWorkflow(workflows.PollAndDispatch)
            w.RegisterWorkflow(workflows.HandleRunner)
            w.RegisterActivity(acts)
            return nil, nil
        },
    })
    if err != nil {
        log.Fatalln(err)
    }
}

shared.RunWorker handles the rest that every worker in the repo needs: tracing, structured logging, Prometheus metrics, and the Temporal client.

It also fits the problem. Each runner gets its own child workflow, so its state, its one-hour ceiling, and its cleanup survive a worker restart without me storing anything. Retries are set per step: polling retries, dispatch never does, and the reap runs even if the workflow is cancelled.

The code

The scaler lives in nomad-temporal-jobs/runnerscaler, next to the other workers in that repo, which include backups, image scanning, and cluster maintenance. The Nomad job files and the Consul KV config are in munchbox.

Originally published at alexfreidah.com.