<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Daniel Schroeder</title>
    <description>The latest articles on DEV Community by Daniel Schroeder (@udondan).</description>
    <link>https://dev.to/udondan</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F258664%2F4896429e-c5cd-45f9-80b3-7c9012ad48d8.webp</url>
      <title>DEV Community: Daniel Schroeder</title>
      <link>https://dev.to/udondan</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/udondan"/>
    <language>en</language>
    <item>
      <title>Orchestrating Renovate, Dependabot, release PRs and tests on main: a priority queue for GitHub Actions</title>
      <dc:creator>Daniel Schroeder</dc:creator>
      <pubDate>Tue, 03 Nov 2026 17:27:21 +0000</pubDate>
      <link>https://dev.to/udondan/orchestrating-renovate-dependabot-release-prs-and-tests-on-main-a-priority-queue-for-github-1cba</link>
      <guid>https://dev.to/udondan/orchestrating-renovate-dependabot-release-prs-and-tests-on-main-a-priority-queue-for-github-1cba</guid>
      <description>&lt;p&gt;A modern repository has four or more parties writing to main, and none of them talks to the others. Developers open pull requests. Renovate or Dependabot open more. A release tool opens one on top. And a workflow runs on main after every merge. Each of them triggers CI, each of them is affected by every merge, and GitHub Actions runs them in whatever order they happen to arrive.&lt;/p&gt;

&lt;p&gt;If CI is cheap and stateless, that's fine. If CI deploys something real, a cloud stack, a staging environment, a database, a Terraform state, it is not fine. Then "whatever order they arrive" means wasted runs at best and two deployments colliding at worst.&lt;/p&gt;

&lt;p&gt;I maintain a few projects where this got out of hand, and I ended up writing a GitHub Action for it: &lt;a href="https://github.com/marketplace/actions/boarding-gate" rel="noopener noreferrer"&gt;&lt;strong&gt;boarding-gate&lt;/strong&gt;&lt;/a&gt;. This article is about the coordination problem it solves, and where I think the same problem lives in your repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four actors, four agendas
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Developers.&lt;/strong&gt; Their PRs are the reason the repository exists. They are also the only ones who get impatient when a run is slow.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Renovate and Dependabot.&lt;/strong&gt; Open several PRs at once, each with auto-merge. By default they rebase every open PR as soon as main moves: Renovate with &lt;code&gt;rebaseWhen: auto&lt;/code&gt;, Dependabot unless &lt;code&gt;rebase-strategy: disabled&lt;/code&gt;. Every rebase is a new commit, every new commit is a new CI run. They don't care whose turn it is.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The release PR.&lt;/strong&gt; release-please, changesets ("Version Packages"), python-semantic-release and friends open a PR that bumps the version and the changelog. It wants to be merged last, because it describes everything merged before it. It is updated the instant main changes, faster than any other PR, so it is reliably the first branch that is up to date and the first run in line.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The pipeline on main.&lt;/strong&gt; Whatever runs on &lt;code&gt;push: main&lt;/code&gt;: the real deployment, the integration test, publishing. It might use the same shared resource as the PR runs, and after every merge it starts immediately.&lt;/p&gt;

&lt;p&gt;The mismatch: the actor that should go last is the fastest, the ones that should go first are the slowest to react, and the bots in the middle produce the most runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The loop
&lt;/h2&gt;

&lt;p&gt;Add branch protection with "Require branches to be up to date before merging" (you should; without it, two individually green PRs can break main together) and the four actors form a loop:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A PR merges.&lt;/li&gt;
&lt;li&gt;The pipeline on main starts.&lt;/li&gt;
&lt;li&gt;The release PR is rebuilt and force-pushed. New run.&lt;/li&gt;
&lt;li&gt;Renovate and Dependabot rebase all their open PRs. One new run each.&lt;/li&gt;
&lt;li&gt;Your other PRs are now behind. You press "Update branch". New run each.&lt;/li&gt;
&lt;li&gt;The first of these runs to finish gets merged. Go to 1.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc69cczcp4ddg76r44kh7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc69cczcp4ddg76r44kh7.png" alt="One merge, four reactions, a new run for every open PR" width="799" height="453"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every merge costs one run per remaining open PR, so merging n PRs costs on the order of n²/2 runs. That's the arithmetic in the Dependabot issue tracker, and it matches what I see. With a 20-minute pipeline and six open PRs, a day's work of merging is a day of waiting.&lt;/p&gt;

&lt;p&gt;The order inside the loop is the real damage. The release PR is rebuilt first, so it wins the next slot and gets tested. Then a bot PR merges, the release PR is behind again, and its run was for nothing. I have watched a release PR go through CI five times in a day and never merge. Meanwhile the fix I actually wanted in was queued behind a linter bump.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why GitHub doesn't coordinate this for you
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;concurrency&lt;/code&gt; groups serialize runs, and since May 2026 &lt;code&gt;queue: max&lt;/code&gt; keeps up to 100 of them pending instead of cancelling all but one. But the queue is first come, first served, and it is blind: it will happily run a branch that is already behind main, and it has no idea that the run it just let through is about to be auto-merged and invalidate everyone behind it.&lt;/p&gt;

&lt;p&gt;GitHub's merge queue is the proper answer to the rebase loop. It tests PRs stacked on each other and merges them itself. Two catches: it exists only for repositories owned by organizations, so nothing on a personal account gets one, and it orders by arrival, not by what the PR is. It also does nothing about the pipeline on main or the deploy that must not overlap.&lt;/p&gt;

&lt;p&gt;The lock actions (&lt;a href="https://github.com/softprops/turnstyle" rel="noopener noreferrer"&gt;softprops/turnstyle&lt;/a&gt;, &lt;a href="https://github.com/ahmadnassri/action-workflow-queue" rel="noopener noreferrer"&gt;ahmadnassri/action-workflow-queue&lt;/a&gt;, &lt;a href="https://github.com/ben-z/gh-action-mutex" rel="noopener noreferrer"&gt;ben-z/gh-action-mutex&lt;/a&gt;, &lt;a href="https://github.com/github/lock" rel="noopener noreferrer"&gt;github/lock&lt;/a&gt;) solve "one at a time" and stop there. The hosted merge queues (Mergify, Trunk, Aviator, Graphite, Kodiak) do have priorities, but they order merges rather than workflow runs, and they are built and priced for organizations.&lt;/p&gt;

&lt;p&gt;None of these let me say: humans first, bots next, release last, main after everyone, one at a time, and never test a branch that can't be merged.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule
&lt;/h2&gt;

&lt;p&gt;Written down, the policy is short:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One run of the workflow at a time. Nothing else touches the stack.&lt;/li&gt;
&lt;li&gt;Order by what the PR is, not when it arrived: developers, then dependency bots, then the release PR, then main.&lt;/li&gt;
&lt;li&gt;An urgent PR can jump the line, including while it is already waiting.&lt;/li&gt;
&lt;li&gt;Never spend a run on a branch that is behind main. Rebase it, or fail it, but don't test it.&lt;/li&gt;
&lt;li&gt;When the run at the front succeeds and its PR will auto-merge, wait for the merge before letting the next one through. Otherwise the next one is stale the moment it starts.&lt;/li&gt;
&lt;li&gt;After a merge, don't let the release PR win just because it rebased fastest. If a higher-priority PR is about to be tested, hold the door.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last rule is the one that took the most iterations to get right, and the one no generic lock can express.&lt;/p&gt;

&lt;h2&gt;
  
  
  boarding-gate
&lt;/h2&gt;

&lt;p&gt;boarding-gate is a composite action that enforces exactly that rule for one workflow. It goes into a first job; every other job &lt;code&gt;needs&lt;/code&gt; it.&lt;/p&gt;

&lt;p&gt;Priorities are label weights. A run's weight is the sum of its PR's label weights, highest first, ties by start time. Runs without a PR (the push to main) get &lt;code&gt;no-pull-request-weight&lt;/code&gt;. Labels are re-read while a run waits, so adding &lt;code&gt;urgent&lt;/code&gt; moves it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F09f7ge99lvbewkprzou5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F09f7ge99lvbewkprzou5.png" alt="Same six runs: arrival order vs. order by label weight" width="799" height="413"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;boarding-gate&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;actions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
      &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
      &lt;span class="na"&gt;pull-requests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
      &lt;span class="na"&gt;statuses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
    &lt;span class="na"&gt;concurrency&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ github.workflow }}-wait-${{ github.event.pull_request.number || github.ref }}&lt;/span&gt;
      &lt;span class="na"&gt;cancel-in-progress&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;udondan/boarding-gate@v1&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;require-up-to-date&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;update-pending-branches&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;update-token&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ secrets.UPDATE_TOKEN }}&lt;/span&gt;
          &lt;span class="na"&gt;status-context&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;boarding-gate&lt;/span&gt;
          &lt;span class="na"&gt;no-pull-request-weight&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;-100&lt;/span&gt;
          &lt;span class="na"&gt;label-weights&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
            &lt;span class="s"&gt;urgent=10&lt;/span&gt;
            &lt;span class="s"&gt;renovate=-5&lt;/span&gt;
            &lt;span class="s"&gt;dependencies=-5&lt;/span&gt;
            &lt;span class="s"&gt;autorelease: pending=-10&lt;/span&gt;

  &lt;span class="na"&gt;deploy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;needs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;boarding-gate&lt;/span&gt;
    &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ !cancelled() }}&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;needs.boarding-gate.result != 'success'&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;exit &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
      &lt;span class="c1"&gt;# the part that touches the shared resource&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What the inputs map to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;require-up-to-date&lt;/code&gt; fails a waiting run as soon as its branch is behind. With &lt;code&gt;update-token&lt;/code&gt; (a PAT or App token; &lt;code&gt;github.token&lt;/code&gt; pushes don't trigger workflows) it rebases the branch instead, and the new commit queues fresh.&lt;/li&gt;
&lt;li&gt;Auto-merge is detected: the turn stays held until the holder's PR is merged or auto-merge gives up (&lt;code&gt;merge-wait-minutes&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;update-pending-branches&lt;/code&gt; is the "hold the door" rule: after a merge, higher-weight PRs with auto-merge that are behind get rebased by the waiting run and go first (&lt;code&gt;yield-to-pending-minutes&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;status-context&lt;/code&gt; writes a commit status with the live position, because composite-action logs don't stream.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;concurrency&lt;/code&gt; group on the wait job cancels a run the moment its PR gets a newer commit. It sits on the wait job only, so a job already using the resource is never cancelled mid-deploy.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9rwpkigllmai7x7zs53g.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9rwpkigllmai7x7zs53g.png" alt="boarding-gate commit statuses: took the turn, position in the queue, waiting for a merge, letting a higher-weight PR go first, branch behind, outdated commit" width="799" height="452"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Under the hood the lock is a git ref: &lt;code&gt;refs/boarding-gate/&amp;lt;workflow&amp;gt;/&amp;lt;n&amp;gt;&lt;/code&gt;. Creating a ref that exists fails atomically, so there is no race and no external service. Forks and Dependabot PRs work; the README covers the &lt;code&gt;workflow_run&lt;/code&gt; companion for forks.&lt;/p&gt;

&lt;p&gt;A reusable workflow with these defaults exists if you'd rather not copy the block into every repository.&lt;/p&gt;

&lt;p&gt;Where that leaves the merge queue: it removes the cost of the rebase loop, so if you have one, let it handle up-to-date branches and auto-merge, and you won't need &lt;code&gt;require-up-to-date&lt;/code&gt; or the pending-branch handling. It still merges in arrival order and still runs the pipeline on main whenever it likes. The ordering, the priorities and the one-at-a-time on main are what boarding-gate adds on top, with or without a merge queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Same rule, other stacks
&lt;/h2&gt;

&lt;p&gt;My shared resource is an AWS CDK test stack. Swap it out and the four actors are still there:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Infrastructure as code.&lt;/strong&gt; Terraform, OpenTofu, Pulumi, CloudFormation. "Error acquiring the state lock" in CI is this problem wearing a different error message. &lt;code&gt;-lock-timeout&lt;/code&gt; makes the second run wait; not starting it until the branch is current is cheaper.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A single staging environment.&lt;/strong&gt; The deploy must not overlap, and the hotfix must go before the dependency bumps. That's &lt;code&gt;urgent=10&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Publishing.&lt;/strong&gt; npm, PyPI, crates.io, Maven Central, GitHub Pages. Whether the release comes from a PR (release-please, changesets) or straight from main (semantic-release, goreleaser, release-it, cargo-release, lerna, nx release, projen), the publishing run and the PR runs share the registry and the version sequence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Database migrations and E2E environments.&lt;/strong&gt; Two migration runs against one database is the same collision as two deploys against one stack.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scarce runners.&lt;/strong&gt; Let build jobs wait at the gate as well. Runs whose branch is already stale never take a runner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Agents.&lt;/strong&gt; Letting coding agents open PRs in parallel produces a bot storm indistinguishable from Renovate's. The same ordering applies, with a label per agent if you want them behind humans.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;boarding-gate queues the runs of one workflow. That's the whole scope. It isn't a merge queue, and it won't predict a merge you click by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Source and README: &lt;a href="https://github.com/udondan/boarding-gate" rel="noopener noreferrer"&gt;github.com/udondan/boarding-gate&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/marketplace/actions/boarding-gate" rel="noopener noreferrer"&gt;GitHub Marketplace listing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Apache-2.0. Bash, bats tests against a fake GitHub API, end-to-end tests in a real repository with bursts of parallel PRs, Renovate auto-merges and relabelling while waiting. Issues and PRs welcome, especially from setups with a shared resource I haven't thought of.&lt;/p&gt;

</description>
      <category>automation</category>
      <category>ci</category>
      <category>devops</category>
      <category>githubactions</category>
    </item>
    <item>
      <title>Your CI is testing the wrong PR first</title>
      <dc:creator>Daniel Schroeder</dc:creator>
      <pubDate>Sun, 01 Nov 2026 12:27:43 +0000</pubDate>
      <link>https://dev.to/udondan/your-ci-is-testing-the-wrong-pr-first-3oa5</link>
      <guid>https://dev.to/udondan/your-ci-is-testing-the-wrong-pr-first-3oa5</guid>
      <description>&lt;p&gt;I maintain a handful of open source projects that deploy real infrastructure in their tests. Not mocks. An actual AWS stack gets created, poked at, torn down. Only one copy of that stack exists, so only one test run can use it at a time.&lt;/p&gt;

&lt;p&gt;That constraint sounds manageable until you add Renovate and release-please to the repo. Suddenly there are six open pull requests on a Tuesday morning, all of them want the stack, and GitHub Actions has no opinion about who should go first. It took me an embarrassingly long time to notice how much of my CI minutes were spent testing commits that were never going to be merged in that state.&lt;/p&gt;

&lt;p&gt;So I wrote &lt;a href="https://github.com/marketplace/actions/boarding-gate" rel="noopener noreferrer"&gt;boarding-gate&lt;/a&gt;, a GitHub Action that queues workflow runs, lets you order them by PR label, and knows a few things about auto-merge and outdated branches that a plain lock doesn't. This post is about the problem first, because I think most people have it without noticing, and the action second.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a normal morning looks like
&lt;/h2&gt;

&lt;p&gt;Say the repo has these PRs open, and branch protection requires branches to be up to date with main before merging (which you want, otherwise you merge things that were never tested together):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Two feature PRs of your own&lt;/li&gt;
&lt;li&gt;Three Renovate PRs with auto-merge enabled&lt;/li&gt;
&lt;li&gt;One release-please PR that bumps the version and changelog&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All six get a CI run when they're opened. Fine, that's expected. Now you merge the first feature PR. Here's what happens next:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The push to main triggers a run on main. Same code that just passed on the PR, give or take a merge commit. That's one wasted run.&lt;/li&gt;
&lt;li&gt;All five remaining PRs are now behind main. release-please rebases its branch immediately. Renovate rebases its three a few minutes later. Your other feature branch waits for you to click "Update branch". Each rebase is a new commit, so each one is a new run. Five more runs.&lt;/li&gt;
&lt;li&gt;The first of those rebased runs that finishes and has auto-merge gets merged. Back to step 1 with four PRs.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For six PRs that works out to something like 6 + 1 + 5 + 1 + 4 + 1 + 3 + 1 + 2 + 1 + 1 = 26 runs to merge six changes. On a normal runner that's just money. On a shared test stack where every run takes 20 minutes and nothing else can run meanwhile, it's the whole afternoon.&lt;/p&gt;

&lt;p&gt;And the order is wrong. The release-please PR rebases fastest, so it tends to be first in line after every merge, even though it's the one PR that explicitly must not be merged until everything else is in. It gets tested, passes, sits there, goes stale on the next merge, gets tested again. I've watched a release PR go through CI five times in a day without ever being merged.&lt;/p&gt;

&lt;h2&gt;
  
  
  "Just use a concurrency group"
&lt;/h2&gt;

&lt;p&gt;That's the first thing everyone says, and it's what I tried first. The GitHub docs even frame &lt;code&gt;concurrency&lt;/code&gt; as the answer to shared environments:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;concurrency&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;integration-tests&lt;/span&gt;
  &lt;span class="na"&gt;cancel-in-progress&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It does make runs go one at a time. What it doesn't do is queue. A concurrency group holds exactly one running job and one pending job. When a third run shows up, it cancels the pending one and takes its place. With six PRs, four of them get cancelled before they ever start, and nothing restarts them. You end up re-running cancelled jobs by hand, which is the thing you were trying to automate away.&lt;/p&gt;

&lt;p&gt;Set &lt;code&gt;cancel-in-progress: true&lt;/code&gt; and it gets worse: the newest run kills whatever is currently using the stack, mid-test, without cleanup.&lt;/p&gt;

&lt;p&gt;And even if it did queue properly, it would be first come, first served. The release-please PR that rebased fastest is still first. There's no knob for "these runs matter less".&lt;/p&gt;

&lt;p&gt;The same goes for the queue actions I found, like &lt;code&gt;softprops/turnstyle&lt;/code&gt; or &lt;code&gt;ben-z/gh-action-mutex&lt;/code&gt;. They're proper queues, so nothing gets cancelled, but they're FIFO and they don't know what a pull request is. A run holding the lock finishes, the lock is free, and the next run in line grabs it even if its branch is already behind main and will have to be retested anyway.&lt;/p&gt;

&lt;p&gt;None of these are bad tools. They solve "one at a time". My problem was "one at a time, in a sensible order, and don't bother testing things that are about to be stale".&lt;/p&gt;

&lt;h2&gt;
  
  
  What about merge queue?
&lt;/h2&gt;

&lt;p&gt;If you can use GitHub's merge queue, you probably should. It fixes the stale branch problem at the root: it tests each PR on top of the ones ahead of it and merges it itself, so there's no rebase-and-retest loop at all. Add a concurrency group on the &lt;code&gt;merge_group&lt;/code&gt; workflow and you also get one at a time.&lt;/p&gt;

&lt;p&gt;Two reasons I couldn't:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Merge queue is only available for repositories owned by an organization. All my projects live under my user account.&lt;/li&gt;
&lt;li&gt;It orders PRs by when they were added to the queue. You can jump the queue by hand, but there's no "release PRs always go last" rule.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So if you're in an org, go read the merge queue docs instead of the rest of this post. If you're not, keep reading.&lt;/p&gt;

&lt;h2&gt;
  
  
  How boarding-gate works
&lt;/h2&gt;

&lt;p&gt;The name is from airports: everyone waits at the gate, and the gate decides who boards next.&lt;/p&gt;

&lt;p&gt;The lock is a git ref. The run that currently has the turn created &lt;code&gt;refs/boarding-gate/&amp;lt;workflow&amp;gt;/&amp;lt;n&amp;gt;&lt;/code&gt;. When it's done, the next run takes the turn by creating ref &lt;code&gt;&amp;lt;n+1&amp;gt;&lt;/code&gt;. Creating a ref that already exists fails, so two runs can't both win. There's no external service, no database, no lease table. Old refs get cleaned up.&lt;/p&gt;

&lt;p&gt;Every run of the workflow starts with a wait job that polls until it's that run's turn. The polling is cheap on purpose: the run that's next in line checks every 30 seconds, everyone further back only every five minutes. With ten runs waiting you're not hammering the API.&lt;/p&gt;

&lt;p&gt;The order of the waiting runs is what makes this different from a mutex:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Each PR label can carry a weight. A run's weight is the sum of its labels. Higher goes first, ties go by start time.&lt;/li&gt;
&lt;li&gt;With &lt;code&gt;renovate=-5&lt;/code&gt; and &lt;code&gt;autorelease: pending=-10&lt;/code&gt;, your own unlabelled PRs (weight 0) are tested first, Renovate updates after that, and the release PR last. Which is exactly when it should run, because by then there's nothing left to merge before it.&lt;/li&gt;
&lt;li&gt;Labels are re-read while waiting. Slap an &lt;code&gt;urgent&lt;/code&gt; label on a PR and its run moves to the front of the queue without re-triggering anything.&lt;/li&gt;
&lt;li&gt;Runs without a PR, like the push to main after a merge, get their own weight. Set it to -100 and main gets tested only once the PRs are done.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The turn is released when the holder finishes, whether it passed, failed or got cancelled. A re-run of the holder keeps the turn. A run that gets cancelled halfway through the job after the wait frees the turn too, so nobody waits on a dead run.&lt;/p&gt;

&lt;p&gt;One more thing a mutex doesn't do: if a PR gets a new commit while its old run is still queued, the old run drops out of the queue and fails as outdated. There's no point testing a commit that's no longer the head of the branch.&lt;/p&gt;

&lt;h2&gt;
  
  
  Setting it up
&lt;/h2&gt;

&lt;p&gt;Add a wait job and make the jobs that touch the shared resource depend on it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;boarding-gate&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;timeout-minutes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;180&lt;/span&gt;
    &lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;actions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
      &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
      &lt;span class="na"&gt;pull-requests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
    &lt;span class="na"&gt;concurrency&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ github.workflow }}-wait-${{ github.event.pull_request.number || github.ref }}&lt;/span&gt;
      &lt;span class="na"&gt;cancel-in-progress&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;udondan/boarding-gate@v1&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;require-up-to-date&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;label-weights&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
            &lt;span class="s"&gt;autorelease: pending=-10&lt;/span&gt;
            &lt;span class="s"&gt;renovate=-5&lt;/span&gt;

  &lt;span class="na"&gt;deploy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;needs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;boarding-gate&lt;/span&gt;
    &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ !cancelled() }}&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Check turn&lt;/span&gt;
        &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;needs.boarding-gate.result != 'success'&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;exit &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
      &lt;span class="c1"&gt;# ... your actual tests against the shared stack&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two bits of that deserve a sentence each, because they look odd.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;concurrency&lt;/code&gt; group on the wait job is per pull request, and it's there to kill the waiting run of a commit that just got replaced by a newer push. Without it, the outdated run sits on a runner until its next poll notices. It groups by PR number rather than branch name, because a fork can have a branch with the same name as one of yours. The group is on the wait job only. Once a run has the turn, the wait job is finished, so the jobs that actually use the stack can never be cancelled by a newer push and always get to clean up.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;Check turn&lt;/code&gt; step exists because a skipped job counts as a passing required check on GitHub. If the wait fails (timeout, outdated branch), &lt;code&gt;deploy&lt;/code&gt; would be skipped, and the PR would show green. So &lt;code&gt;deploy&lt;/code&gt; runs anyway and fails itself in its first step.&lt;/p&gt;

&lt;p&gt;All runs of the workflow have to go through the wait job. A run that skips it isn't in the queue and will happily use the stack while someone else has the turn.&lt;/p&gt;

&lt;p&gt;If you'd rather not copy that block into every repo, there's also a reusable workflow that bundles the same job with sensible defaults (&lt;code&gt;urgent=10&lt;/code&gt; included):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;boarding-gate&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;actions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
      &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
      &lt;span class="na"&gt;pull-requests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
      &lt;span class="na"&gt;statuses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
    &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;udondan/boarding-gate/.github/workflows/boarding-gate.yml@v1&lt;/span&gt;
    &lt;span class="na"&gt;secrets&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;update-token&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ secrets.UPDATE_TOKEN }}&lt;/span&gt;

  &lt;span class="na"&gt;deploy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;needs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;boarding-gate&lt;/span&gt;
    &lt;span class="c1"&gt;# ...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The part that took the most iterations: stale branches and auto-merge
&lt;/h2&gt;

&lt;p&gt;Ordering by label was the easy part. Most of the commits in the repo went into the interaction between the queue, branch protection and auto-merge, because that's where the wasted runs actually come from.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Don't test a branch that's behind.&lt;/strong&gt; With &lt;code&gt;require-up-to-date: true&lt;/code&gt;, a waiting run checks whether its branch is still up to date with main, and fails immediately if it isn't. Not after the test, not after taking the turn. It's going to have to be rebased and retested anyway, so there's no point letting it hold the stack for 20 minutes first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rebase it if nobody else will.&lt;/strong&gt; Some branches never get updated on their own. release-please leaves its PR alone if the merged commit didn't touch the changelog, so its run would fail as outdated forever. Pass an &lt;code&gt;update-token&lt;/code&gt; (a PAT or App token, because pushes made with &lt;code&gt;github.token&lt;/code&gt; don't trigger workflows) and the action rebases the branch onto main before failing. The new commit starts a fresh run that's actually worth running.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hold the turn through the merge.&lt;/strong&gt; This one bit me. A Renovate PR with auto-merge passes CI, and about three seconds later GitHub merges it. In those three seconds the turn is free, the next run in line takes it, checks that its branch is up to date (it is, for now), and starts testing. Then the merge lands, and it's testing a branch that's already behind. So when the holder succeeded and its PR has auto-merge enabled, the turn stays held until the PR is actually merged or closed, or it's clear the merge isn't coming: auto-merge got disabled, a required check failed, the branch fell behind or got a new commit. &lt;code&gt;merge-wait-minutes&lt;/code&gt; caps the wait in case a review is missing. Once it's merged, the waiting runs fail as outdated right away instead of after a full test.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Let the right PR go first after a merge.&lt;/strong&gt; After a merge, PRs get updated at different speeds. release-please rebases within seconds, Renovate takes a few minutes. So for a short while the release PR is the only run in the queue, and it takes the turn even though it has the lowest weight. The action now checks whether there are open auto-merge PRs with a higher weight whose head commit hasn't been tested yet, and yields to them for up to &lt;code&gt;yield-to-pending-minutes&lt;/code&gt;. With &lt;code&gt;update-pending-branches: true&lt;/code&gt; it goes one step further and rebases those higher-weight PRs itself, so the order by weight holds across merges without waiting for Renovate's schedule.&lt;/p&gt;

&lt;p&gt;With all of that on, the morning from earlier looks different. One run per PR per merge of something ahead of it, in weight order, main tested last, and the release PR tested exactly once, right before it's the only thing left.&lt;/p&gt;

&lt;h2&gt;
  
  
  Seeing where you are in the queue
&lt;/h2&gt;

&lt;p&gt;Composite actions don't stream their logs, you only see them once the step is done. Which is annoying for a step whose whole job is to wait. So the action can set a commit status on the PR (&lt;code&gt;status-context: boarding-gate&lt;/code&gt;, needs &lt;code&gt;statuses: write&lt;/code&gt;) that updates live: &lt;code&gt;Position 2 of 4 (weight -5). Turn held by #123.&lt;/code&gt; It turns green when the run takes the turn and red if the wait failed. You can see at a glance why nothing is happening on your PR without opening a single workflow log.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fef8ko9qu790pof6ftqbb.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fef8ko9qu790pof6ftqbb.webp" alt=" " width="799" height="452"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Pull requests from forks and Dependabot
&lt;/h2&gt;

&lt;p&gt;The first version of this post told you to skip the wait job for these. That's no longer necessary.&lt;/p&gt;

&lt;p&gt;A run of a PR from a fork gets a read-only token, so it can't create the lock ref. It now waits in the queue anyway, ordered by its labels like any other run. When it's first in line and the turn is free, a second run takes the turn on its behalf. That run comes from a small &lt;code&gt;workflow_run&lt;/code&gt; workflow you add next to the one that waits. &lt;code&gt;workflow_run&lt;/code&gt; runs the workflow from your default branch with a writable token, also for runs of forks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Priority boarding for forks&lt;/span&gt;

&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;workflow_run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="c1"&gt;# zizmor: ignore[dangerous-triggers] no code of the pull request is run&lt;/span&gt;
    &lt;span class="na"&gt;workflows&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;Test&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;types&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;in_progress&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;

&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{}&lt;/span&gt;

&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;boarding-gate&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;github.event.workflow_run.event == 'pull_request' &amp;amp;&amp;amp; github.event.workflow_run.head_repository.full_name != github.repository&lt;/span&gt;
    &lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;actions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
      &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
      &lt;span class="na"&gt;pull-requests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
      &lt;span class="na"&gt;statuses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;write&lt;/span&gt;
    &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;udondan/boarding-gate/.github/workflows/boarding-gate.yml@v1&lt;/span&gt;
    &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;run-id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ format('{0}', github.event.workflow_run.id) }}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It waits in the place of the fork's run, creates the lock naming that run, sets the commit status the fork's run can't set itself, and ends. The turn belongs to the run named in the lock, not the one that created it, so the fork's run sees its name and carries on with its tests. It never checks out the PR's code, so the writable token isn't exposed to it. If you forget to add the workflow, the fork's run gives up after five minutes instead of holding up everyone behind it.&lt;/p&gt;

&lt;p&gt;Forks still get no secrets on &lt;code&gt;pull_request&lt;/code&gt;, so only jobs that work without them can run, and their branches are never rebased. On a self-hosted runner that gets reused, a read-only token doesn't make the fork's code safe: it can stay on the machine and read the credentials of later jobs. Run forks on GitHub-hosted or throwaway runners.&lt;/p&gt;

&lt;p&gt;Dependabot PRs need nothing extra. Their token is read-only by default, but the &lt;code&gt;permissions&lt;/code&gt; of the wait job raise it. They only see Dependabot secrets, so store &lt;code&gt;update-token&lt;/code&gt; there as well. And if the token can't write after all, the action fails right away instead of blocking the queue until the timeout.&lt;/p&gt;

&lt;p&gt;Workflows on &lt;code&gt;pull_request_target&lt;/code&gt; work too. Their token is writable even for forks, so they take the turn themselves and need no &lt;code&gt;workflow_run&lt;/code&gt; workflow. The usual warning applies: those runs have your secrets, so the jobs after the wait must not run the PR's code. For forks, &lt;code&gt;pull_request&lt;/code&gt; with the workflow above is the safer choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things it can't do
&lt;/h2&gt;

&lt;p&gt;It can't predict a manual merge. If you click the merge button yourself while a run is between "turn is free" and "branch is up to date", that run will test a branch that's about to be stale. Auto-merge is the predictable path.&lt;/p&gt;

&lt;p&gt;And it's not a merge queue. If you have one, use it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wrapping up
&lt;/h2&gt;

&lt;p&gt;It's a composite action written in bash, Apache-2.0, tested with bats against a fake GitHub API and end-to-end in a real repo with bursts of parallel PRs, relabelling, Renovate auto-merges, pushes to main, cancelled and re-run runs, and PRs from a fork. The repo is at &lt;a href="https://github.com/udondan/boarding-gate" rel="noopener noreferrer"&gt;github.com/udondan/boarding-gate&lt;/a&gt; and it's on the &lt;a href="https://github.com/marketplace/actions/boarding-gate" rel="noopener noreferrer"&gt;Marketplace&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If your CI has a resource that only one run can use at a time, and you've got bots opening PRs, I'd be curious whether it fits your setup or where it falls over. Issues welcome.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>githubactions</category>
      <category>opensource</category>
      <category>cicd</category>
    </item>
    <item>
      <title>avanti: One YAML Spec, Files from Anywhere</title>
      <dc:creator>Daniel Schroeder</dc:creator>
      <pubDate>Thu, 07 May 2026 11:27:55 +0000</pubDate>
      <link>https://dev.to/udondan/avanti-one-yaml-spec-files-from-anywhere-l81</link>
      <guid>https://dev.to/udondan/avanti-one-yaml-spec-files-from-anywhere-l81</guid>
      <description>&lt;p&gt;You have a &lt;code&gt;renovate.json&lt;/code&gt; in 30 repos. A platform team owns the canonical version. Six months later, half those repos are running a config from March that nobody updated. One team added a custom rule. Another deleted a section by accident. Nobody knows which version is "right" anymore.&lt;/p&gt;

&lt;p&gt;This is config drift. By the time anyone looks, the configs have diverged.&lt;/p&gt;

&lt;h2&gt;
  
  
  What avanti does
&lt;/h2&gt;

&lt;p&gt;avanti is a CLI tool that pulls local files from a declarative YAML spec. You describe what each file should contain and where it comes from. Then you run &lt;code&gt;avanti pull&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It has two commands:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;avanti diff&lt;/code&gt; — shows a colored git-diff-style preview of what would change, exits &lt;code&gt;1&lt;/code&gt; if changes exist&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;avanti pull&lt;/code&gt; — fetches everything, shows the diff, and asks for confirmation before writing&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No daemon, no server, no agent. Just a spec file and two commands.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;p&gt;A &lt;code&gt;src&lt;/code&gt; value can be an HTTP URL, a local path, a GitHub file, a GitLab file, a shell command, or inline raw content. You pick per entry.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;files&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="c1"&gt;# HTTP&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https://example.com/base-config.yml&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;base-config.yml&lt;/span&gt;

  &lt;span class="c1"&gt;# Local path&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;~/shared/scripts/deploy.sh&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;scripts/deploy.sh&lt;/span&gt;
    &lt;span class="na"&gt;mode&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;0755'&lt;/span&gt;

  &lt;span class="c1"&gt;# GitHub&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/standards&lt;/span&gt;
        &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;eslint.config.js&lt;/span&gt;
        &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;

  &lt;span class="c1"&gt;# Shell command&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;exec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;aws ssm get-parameter --name /app/db-config --with-decryption --query Parameter.Value --output text&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;config/db.json&lt;/span&gt;
    &lt;span class="na"&gt;mode&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;0600'&lt;/span&gt;

  &lt;span class="c1"&gt;# Inline content&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;raw&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
        &lt;span class="s"&gt;# THIS FILE IS MANAGED — run `avanti pull` to update&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;header.txt&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can also combine multiple sources into one file by passing &lt;code&gt;src&lt;/code&gt; as a list. They're fetched in order and joined with a newline.&lt;/p&gt;

&lt;h2&gt;
  
  
  Variables
&lt;/h2&gt;

&lt;p&gt;Define reusable values once, reference them everywhere with &lt;code&gt;$name&lt;/code&gt;. Environment variables use &lt;code&gt;$env:NAME&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;variables&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;standards_ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v2.4.1&lt;/span&gt;
  &lt;span class="na"&gt;region&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;eu-west-1&lt;/span&gt;

&lt;span class="na"&gt;files&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/standards&lt;/span&gt;
        &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;renovate.json&lt;/span&gt;
        &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$standards_ref&lt;/span&gt;

  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/infra&lt;/span&gt;
        &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;k8s/deployment-template.yaml&lt;/span&gt;
        &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$env:DEPLOY_VERSION&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;k8s/deployment.yaml&lt;/span&gt;
    &lt;span class="na"&gt;replace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;{REGION}'&lt;/span&gt;
        &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$region&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;{ENV}'&lt;/span&gt;
        &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$env:ENVIRONMENT&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Bump &lt;code&gt;standards_ref&lt;/code&gt; in one place. Run &lt;code&gt;avanti diff&lt;/code&gt;. See every file that will change before you touch anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  Post-processing
&lt;/h2&gt;

&lt;p&gt;Each entry supports a &lt;code&gt;replace&lt;/code&gt; list for string or regex substitutions, and a &lt;code&gt;post&lt;/code&gt; field to pipe content through a shell script. Both run after fetching, before writing.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https://example.com/template.yml&lt;/span&gt;
  &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;config.yml&lt;/span&gt;
  &lt;span class="na"&gt;replace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;{EMAIL}'&lt;/span&gt;
      &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$email&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/\d+\.\d+\.\d+/&lt;/span&gt;
      &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$env:APP_VERSION&lt;/span&gt;
  &lt;span class="na"&gt;post&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;yq e '.metadata.name = "my-app"' -&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Use case 1: A global CLAUDE.md your whole team actually shares
&lt;/h2&gt;

&lt;p&gt;Per-project &lt;code&gt;CLAUDE.md&lt;/code&gt; files should contain project-specific context. But company-wide coding standards, team conventions, and security guidelines don't belong there — they belong in a global &lt;code&gt;~/.claude/CLAUDE.md&lt;/code&gt; that every developer on the team carries. The problem is keeping that file in sync across 20 machines.&lt;/p&gt;

&lt;p&gt;avanti solves this by assembling the global file from central sources. Each developer has a personal &lt;code&gt;.avanti.yml&lt;/code&gt; (or the team ships one via onboarding). Running &lt;code&gt;avanti pull&lt;/code&gt; rebuilds the file from the current canonical versions.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;variables&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;team&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;backend&lt;/span&gt;
  &lt;span class="na"&gt;oncall_channel&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;#backend-oncall'&lt;/span&gt;

&lt;span class="na"&gt;files&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;raw&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;# AI Assistant Guidelines&lt;/span&gt;
          &lt;span class="s"&gt;&amp;lt;!-- THIS FILE IS MANAGED — run `avanti pull` to update --&amp;gt;&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;gitlab&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;project&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;platform/ai-standards&lt;/span&gt;
          &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;teams/backend-rules.md&lt;/span&gt;
          &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/shared-prompts&lt;/span&gt;
          &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;company-standards.md&lt;/span&gt;
          &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;raw&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;## Team Context&lt;/span&gt;
          &lt;span class="s"&gt;Team: $team&lt;/span&gt;
          &lt;span class="s"&gt;Oncall: $oncall_channel&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;~/.claude/CLAUDE.md&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The platform team updates &lt;code&gt;backend-rules.md&lt;/code&gt; once. Every developer who runs &lt;code&gt;avanti pull&lt;/code&gt; gets it. No Slack message asking which version is current, no six-month-old guidelines silently telling the AI the wrong thing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use case 2: Pinned shared tooling configs
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;variables&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;standards_ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v2.4.1&lt;/span&gt;

&lt;span class="na"&gt;files&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/standards&lt;/span&gt;
        &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;renovate.json&lt;/span&gt;
        &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$standards_ref&lt;/span&gt;

  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/standards&lt;/span&gt;
        &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;eslint.config.js&lt;/span&gt;
        &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$standards_ref&lt;/span&gt;

  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/standards&lt;/span&gt;
        &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tsconfig.base.json&lt;/span&gt;
        &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;$standards_ref&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All files pinned to &lt;code&gt;v2.4.1&lt;/code&gt;. When the platform team cuts &lt;code&gt;v2.5.0&lt;/code&gt;, projects bump one variable, diff, and apply. No manual file hunting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use case 3: Drift detection in CI
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;files&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;raw&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;# THIS FILE IS MANAGED — run `avanti pull` to update&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;github&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;repo&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;org/ci-templates&lt;/span&gt;
          &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;workflows/security-scan.yml&lt;/span&gt;
          &lt;span class="na"&gt;ref&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;.github/workflows/security-scan.yml&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Add &lt;code&gt;avanti diff&lt;/code&gt; as a CI step. If a developer edited the managed file by hand, the pipeline fails. The fix is to run &lt;code&gt;avanti pull&lt;/code&gt;, not to argue about whose edit was right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Atomic writes
&lt;/h2&gt;

&lt;p&gt;avanti stages all files to a temp directory first. If any source fails, nothing gets written. You either get a full successful update or nothing changes. No half-applied state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Working directory and path safety
&lt;/h2&gt;

&lt;p&gt;All &lt;code&gt;src&lt;/code&gt; and &lt;code&gt;target&lt;/code&gt; paths resolve relative to the working directory, not the config file's location. This lets you use one shared config across many projects:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="nb"&gt;dir &lt;/span&gt;&lt;span class="k"&gt;in &lt;/span&gt;services/&lt;span class="k"&gt;*&lt;/span&gt;/&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;avanti &lt;span class="nt"&gt;-c&lt;/span&gt; shared/avanti.yml &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; pull &lt;span class="nt"&gt;--yes&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Target paths cannot escape the working directory. A &lt;code&gt;target&lt;/code&gt; of &lt;code&gt;../../etc/passwd&lt;/code&gt; is an error. Absolute targets are only allowed when the working directory is &lt;code&gt;/&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Install
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-g&lt;/span&gt; @udondan/avanti
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or run without installing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @udondan/avanti &lt;span class="nt"&gt;--help&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Drop a &lt;code&gt;.avanti.yml&lt;/code&gt; in your project root, define your files, and run &lt;code&gt;avanti diff&lt;/code&gt; to see what it would do.&lt;/p&gt;

&lt;p&gt;The config format is in the README. The source is on GitHub at udondan/scync. Issues and PRs are open.&lt;/p&gt;

&lt;p&gt;If you've handled this differently, I'm curious what you're doing.&lt;/p&gt;

</description>
      <category>tooling</category>
      <category>ai</category>
      <category>cli</category>
      <category>devtools</category>
    </item>
    <item>
      <title>Running Docker MCP Gateway on Linux (Without Docker Desktop)</title>
      <dc:creator>Daniel Schroeder</dc:creator>
      <pubDate>Thu, 09 Apr 2026 18:46:30 +0000</pubDate>
      <link>https://dev.to/udondan/running-docker-mcp-gateway-on-linux-without-docker-desktop-4da2</link>
      <guid>https://dev.to/udondan/running-docker-mcp-gateway-on-linux-without-docker-desktop-4da2</guid>
      <description>&lt;p&gt;Docker's MCP Toolkit is a great way to expose Model Context Protocol servers to AI clients like Claude, n8n, or Cursor. Out of the box it's designed for Docker Desktop on macOS and Windows — but what if you want to run it on a headless Linux server? A Raspberry Pi, a VPS, a home lab box?&lt;/p&gt;

&lt;p&gt;This guide walks through setting it up from scratch on Linux (Debian/Ubuntu, arm64 or amd64), including secrets management, custom MCP server images, and a systemd service that starts automatically on boot.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This guide is based on getting it actually working on a Raspberry Pi 5 running&lt;br&gt;
Debian 13. Several things that look like they should work on Linux don't — I'll&lt;br&gt;
call those out explicitly so you don't waste time on the same dead ends.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  What We're Building
&lt;/h2&gt;

&lt;p&gt;A self-hosted MCP gateway that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Runs any MCP server as a Docker container&lt;/li&gt;
&lt;li&gt;Exposes them all behind a single HTTP endpoint with Bearer token auth&lt;/li&gt;
&lt;li&gt;Starts automatically on boot via systemd
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AI Client  →  http://your-server:8811/sse  →  docker-mcp gateway  →  MCP containers
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Linux host with Docker installed (Docker Engine, not Docker Desktop)&lt;/li&gt;
&lt;li&gt;Your user in the &lt;code&gt;docker&lt;/code&gt; group (&lt;code&gt;sudo usermod -aG docker $USER&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;SSH access if setting up remotely&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Step 1 — Install the docker-mcp Binary
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;docker mcp&lt;/code&gt; command is a Docker CLI plugin. On Linux you install it directly from the GitHub releases — no Docker Desktop needed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Create the CLI plugins directory&lt;/span&gt;
&lt;span class="nb"&gt;sudo mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; /usr/local/lib/docker/cli-plugins

&lt;span class="c"&gt;# Download the binary for your architecture&lt;/span&gt;
&lt;span class="c"&gt;# arm64 (Raspberry Pi 4/5, Apple Silicon VMs):&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  https://github.com/docker/mcp-gateway/releases/download/v0.41.0/docker-mcp-linux-arm64.tar.gz &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;sudo tar&lt;/span&gt; &lt;span class="nt"&gt;-xz&lt;/span&gt; &lt;span class="nt"&gt;-C&lt;/span&gt; /usr/local/lib/docker/cli-plugins/

&lt;span class="c"&gt;# amd64 (regular x86 server):&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  https://github.com/docker/mcp-gateway/releases/download/v0.41.0/docker-mcp-linux-amd64.tar.gz &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;sudo tar&lt;/span&gt; &lt;span class="nt"&gt;-xz&lt;/span&gt; &lt;span class="nt"&gt;-C&lt;/span&gt; /usr/local/lib/docker/cli-plugins/

&lt;span class="nb"&gt;sudo chmod&lt;/span&gt; +x /usr/local/lib/docker/cli-plugins/docker-mcp

&lt;span class="c"&gt;# Verify&lt;/span&gt;
docker mcp &lt;span class="nt"&gt;--version&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check the &lt;a href="https://github.com/docker/mcp-gateway/releases" rel="noopener noreferrer"&gt;releases page&lt;/a&gt; for the latest version.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 2 — The docker-pass Workaround
&lt;/h2&gt;

&lt;p&gt;This is the first Linux-specific gotcha. &lt;code&gt;docker mcp&lt;/code&gt; CLI commands (like &lt;code&gt;docker mcp server ls&lt;/code&gt;) expect a Docker CLI plugin named &lt;code&gt;docker-pass&lt;/code&gt;. This binary ships with Docker Desktop on macOS but not on Linux, causing this error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;docker pass has not been installed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fix: a small wrapper script that satisfies the Docker CLI plugin protocol and delegates to &lt;code&gt;docker-credential-pass&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;First, install &lt;code&gt;docker-credential-pass&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# arm64:&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  https://github.com/docker/docker-credential-helpers/releases/download/v0.9.5/docker-credential-pass-v0.9.5.linux-arm64 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-o&lt;/span&gt; /usr/local/bin/docker-credential-pass

&lt;span class="c"&gt;# amd64:&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  https://github.com/docker/docker-credential-helpers/releases/download/v0.9.5/docker-credential-pass-v0.9.5.linux-amd64 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-o&lt;/span&gt; /usr/local/bin/docker-credential-pass

&lt;span class="nb"&gt;sudo chmod&lt;/span&gt; +x /usr/local/bin/docker-credential-pass
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then create the wrapper plugin:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo tee&lt;/span&gt; /usr/local/lib/docker/cli-plugins/docker-pass &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /dev/null &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
#!/bin/bash
if [[ "&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="sh"&gt;" == "docker-cli-plugin-metadata" ]]; then
  echo '{"SchemaVersion":"0.1.0","Vendor":"Docker","Version":"v1.0.0","ShortDescription":"Docker Pass secrets helper"}'
  exit 0
fi
exec docker-credential-pass "&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="sh"&gt;"
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="nb"&gt;sudo chmod&lt;/span&gt; +x /usr/local/lib/docker/cli-plugins/docker-pass
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verify Docker recognizes it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker info &lt;span class="nt"&gt;--format&lt;/span&gt; &lt;span class="s1"&gt;'{{.ClientInfo.Plugins}}'&lt;/span&gt; | &lt;span class="nb"&gt;tr&lt;/span&gt; &lt;span class="s1"&gt;','&lt;/span&gt; &lt;span class="s1"&gt;'\n'&lt;/span&gt; | &lt;span class="nb"&gt;grep &lt;/span&gt;pass
&lt;span class="c"&gt;# Should show: ...pass /usr/local/lib/docker/cli-plugins/docker-pass...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: This wrapper is only needed for &lt;code&gt;docker mcp&lt;/code&gt; CLI commands. The gateway&lt;br&gt;
itself uses a different mechanism for secrets — covered in Step 5.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Step 3 — Pull or Load Your MCP Images
&lt;/h2&gt;

&lt;p&gt;Any Docker image that implements the MCP stdio protocol can be used as an MCP server.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option A: Images from the official Docker MCP catalog
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker pull mcp/playwright:latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Option B: Custom/private images
&lt;/h3&gt;

&lt;p&gt;Transfer from another machine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# On the source machine:&lt;/span&gt;
docker save mcp/my-server:latest | ssh your-linux-host &lt;span class="s2"&gt;"docker load"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Step 4 — Configure the Gateway
&lt;/h2&gt;

&lt;p&gt;Create the config directory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; ~/.docker/mcp/catalogs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  registry.yaml — which servers to enable
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; ~/.docker/mcp/registry.yaml &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
registry:
  playwright:
    ref: ""
  my-server:
    ref: ""
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  catalog.json — where to find catalog definitions
&lt;/h3&gt;

&lt;p&gt;The gateway needs to know where your catalog files live. By default it only reads the official Docker MCP catalog. Register additional catalogs here:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; ~/.docker/mcp/catalog.json &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
{
  "catalogs": {
    "docker-mcp": {
      "displayName": "Docker MCP Catalog",
      "url": "https://desktop.docker.com/mcp/catalog/v2/catalog.yaml"
    },
    "my-catalog": {
      "displayName": "My Custom Servers",
      "url": "/home/youruser/.docker/mcp/catalogs/my-catalog.yaml"
    }
  }
}
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  A catalog YAML for a custom server
&lt;/h3&gt;

&lt;p&gt;The catalog defines how a server runs and which env vars it needs. List &lt;strong&gt;all&lt;/strong&gt; env vars under &lt;code&gt;secrets:&lt;/code&gt; — including non-sensitive ones like usernames.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Linux gotcha&lt;/strong&gt;: The &lt;code&gt;config:&lt;/code&gt; field in catalog YAMLs is not used for env var&lt;br&gt;
injection on Linux. Everything must be in &lt;code&gt;secrets:&lt;/code&gt; to be passed to containers.&lt;br&gt;
&lt;/p&gt;
&lt;/blockquote&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;registry&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;my-server&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;My MCP Server&lt;/span&gt;
    &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Does something useful&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;mcp/my-server:latest&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;server&lt;/span&gt;
    &lt;span class="na"&gt;tools&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[]&lt;/span&gt;
    &lt;span class="na"&gt;secrets&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;my-server.api_key&lt;/span&gt;
        &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;API_KEY&lt;/span&gt;
        &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;API key for the service&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;my-server.username&lt;/span&gt;
        &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;USERNAME&lt;/span&gt;
        &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Your username&lt;/span&gt;
&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;my-catalog&lt;/span&gt;
&lt;span class="na"&gt;displayName&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;My Catalog&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Step 5 — Secrets
&lt;/h2&gt;

&lt;p&gt;This is the second major Linux gotcha. &lt;code&gt;docker mcp&lt;/code&gt; uses a secrets engine (&lt;code&gt;se://&lt;/code&gt; URIs) that is Docker Desktop-only and doesn't work on Linux. The &lt;code&gt;docker mcp secret set&lt;/code&gt; command will fail with &lt;code&gt;docker pass has not been installed&lt;/code&gt; even after you install the wrapper from Step 2.&lt;/p&gt;

&lt;p&gt;The solution is the &lt;code&gt;--secrets&lt;/code&gt; flag, which points the gateway at a plain env file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Create the secrets file&lt;/span&gt;
&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; ~/.docker/mcp/secrets.env &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;'
my-server.api_key=your-api-key-here
my-server.username=your-username
&lt;/span&gt;&lt;span class="no"&gt;EOF

&lt;/span&gt;&lt;span class="c"&gt;# Restrict permissions — only your user can read it&lt;/span&gt;
&lt;span class="nb"&gt;chmod &lt;/span&gt;600 ~/.docker/mcp/secrets.env
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key names map to the &lt;code&gt;name&lt;/code&gt; field in the catalog's &lt;code&gt;secrets:&lt;/code&gt; list.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is this secure?&lt;/strong&gt; The file is &lt;code&gt;chmod 600&lt;/code&gt; — readable only by your user, same as &lt;code&gt;~/.ssh/id_rsa&lt;/code&gt;. Anyone who can read it already has root or is you. If you want GPG encryption at rest, you can store sensitive values in &lt;code&gt;pass&lt;/code&gt; and populate the file from it — but for an unattended service the threat model is the same either way.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are secrets isolated between MCP servers?&lt;/strong&gt; Yes, completely. The secrets file is never passed to or mounted into any container. The gateway reads it internally and uses it purely as a lookup table. When spawning each container it passes only the specific &lt;code&gt;-e VAR=value&lt;/code&gt; flags declared in that server's catalog &lt;code&gt;secrets:&lt;/code&gt; list. You can verify this with &lt;code&gt;--dry-run --verbose&lt;/code&gt; — the &lt;code&gt;docker run&lt;/code&gt; command for each server is logged in full, and you'll see that playwright gets zero secret env vars, while my-server only gets &lt;code&gt;USERNAME&lt;/code&gt; and &lt;code&gt;API_KEY&lt;/code&gt;. There is no way for one MCP server to access another's credentials.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 6 — Test the Gateway
&lt;/h2&gt;

&lt;p&gt;Do a dry run first:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker mcp gateway run &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--dry-run&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--verbose&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--secrets&lt;/span&gt; ~/.docker/mcp/secrets.env &lt;span class="se"&gt;\&lt;/span&gt;
  2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should see all your configured servers listed and their tools counted, with no &lt;code&gt;Warning: Secret '...' not found&lt;/code&gt; lines. If warnings appear, check that the key names in &lt;code&gt;secrets.env&lt;/code&gt; exactly match the &lt;code&gt;name&lt;/code&gt; fields in your catalog YAML.&lt;/p&gt;

&lt;p&gt;If everything looks good, start it live:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker mcp gateway run &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--transport&lt;/span&gt; sse &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--port&lt;/span&gt; 8811 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--secrets&lt;/span&gt; ~/.docker/mcp/secrets.env
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Step 7 — systemd Service
&lt;/h2&gt;

&lt;p&gt;Create the service file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo tee&lt;/span&gt; /etc/systemd/system/mcp-gateway.service &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /dev/null &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;
[Unit]
Description=Docker MCP Gateway
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=simple
User=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;whoami&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;
Environment=HOME=&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="sh"&gt;
ExecStart=/usr/local/lib/docker/cli-plugins/docker-mcp gateway run &lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="sh"&gt;
  --transport sse &lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="sh"&gt;
  --port 8811 &lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="sh"&gt;
  --secrets &lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="sh"&gt;/.docker/mcp/secrets.env
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Set a stable Bearer token that survives restarts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo mkdir&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; /etc/systemd/system/mcp-gateway.service.d

&lt;span class="nv"&gt;TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;openssl rand &lt;span class="nt"&gt;-hex&lt;/span&gt; 32&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Save this token: &lt;/span&gt;&lt;span class="nv"&gt;$TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;span class="nb"&gt;sudo tee&lt;/span&gt; /etc/systemd/system/mcp-gateway.service.d/token.conf &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /dev/null &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;
[Service]
Environment=MCP_GATEWAY_AUTH_TOKEN=&lt;/span&gt;&lt;span class="nv"&gt;$TOKEN&lt;/span&gt;&lt;span class="sh"&gt;
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without this, the gateway generates a new random token on every start — which means reconfiguring every client after each restart.&lt;/p&gt;

&lt;p&gt;Enable and start:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl daemon-reload
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl &lt;span class="nb"&gt;enable &lt;/span&gt;mcp-gateway.service
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl start mcp-gateway.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check it's running:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl status mcp-gateway.service
journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; mcp-gateway.service &lt;span class="nt"&gt;-f&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Connecting a Client
&lt;/h2&gt;

&lt;p&gt;The gateway runs on port &lt;code&gt;8811&lt;/code&gt; with SSE transport:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;URL:   http://your-server:8811/sse
Auth:  Authorization: Bearer &amp;lt;your-token&amp;gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Claude Desktop
&lt;/h3&gt;

&lt;p&gt;Add to &lt;code&gt;~/Library/Application Support/Claude/claude_desktop_config.json&lt;/code&gt;&lt;br&gt;
(macOS) or the equivalent on your OS:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"mcpServers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"my-gateway"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"npx"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"args"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"mcp-remote"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"http://your-server:8811/sse"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"--header"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"Authorization: Bearer &amp;lt;your-token&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"--allow-http"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"--transport"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"sse-only"&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two flags are required here that aren't obvious:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;--allow-http&lt;/code&gt; — &lt;code&gt;mcp-remote&lt;/code&gt; blocks non-HTTPS URLs by default&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--transport sse-only&lt;/code&gt; — the default &lt;code&gt;http-first&lt;/code&gt; strategy sends a POST that
the gateway rejects with &lt;code&gt;sessionid must be provided&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Troubleshooting
&lt;/h2&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;docker pass has not been installed&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;docker-pass&lt;/code&gt; CLI plugin wrapper is missing or not executable. Re-check Step 2.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;Warning: Secret '...' not found&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The key name in &lt;code&gt;secrets.env&lt;/code&gt; doesn't match the &lt;code&gt;name&lt;/code&gt; field in the catalog YAML, or you forgot to pass &lt;code&gt;--secrets&lt;/code&gt; to the gateway. Check with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker mcp gateway run &lt;span class="nt"&gt;--dry-run&lt;/span&gt; &lt;span class="nt"&gt;--verbose&lt;/span&gt; &lt;span class="nt"&gt;--secrets&lt;/span&gt; ~/.docker/mcp/secrets.env 2&amp;gt;&amp;amp;1 | &lt;span class="nb"&gt;grep &lt;/span&gt;Warning
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Server shows 0 tools in dry-run but works live
&lt;/h3&gt;

&lt;p&gt;Some servers need the actual secrets present to respond to tool listing. This is normal — the gateway still starts them correctly at runtime.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;cannot use --port with --transport=stdio&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;You must specify &lt;code&gt;--transport sse&lt;/code&gt; when using &lt;code&gt;--port&lt;/code&gt;. The default transport is stdio (for direct client connections), not HTTP.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;sessionid must be provided&lt;/code&gt; in mcp-remote
&lt;/h3&gt;

&lt;p&gt;Add &lt;code&gt;--transport sse-only&lt;/code&gt; to the &lt;code&gt;mcp-remote&lt;/code&gt; args. The default transport strategy tries Streamable HTTP first, which the gateway doesn't support.&lt;/p&gt;




&lt;h2&gt;
  
  
  Keeping Things Updated
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Upgrade docker-mcp
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;VERSION&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;v0.41.0  &lt;span class="c"&gt;# replace with latest&lt;/span&gt;
&lt;span class="nv"&gt;ARCH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;arm64       &lt;span class="c"&gt;# or amd64&lt;/span&gt;

&lt;span class="nb"&gt;sudo &lt;/span&gt;curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  https://github.com/docker/mcp-gateway/releases/download/&lt;span class="nv"&gt;$VERSION&lt;/span&gt;/docker-mcp-linux-&lt;span class="nv"&gt;$ARCH&lt;/span&gt;.tar.gz &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;sudo tar&lt;/span&gt; &lt;span class="nt"&gt;-xz&lt;/span&gt; &lt;span class="nt"&gt;-C&lt;/span&gt; /usr/local/lib/docker/cli-plugins/

&lt;span class="nb"&gt;sudo chmod&lt;/span&gt; +x /usr/local/lib/docker/cli-plugins/docker-mcp
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart mcp-gateway.service
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Update a custom MCP image
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker save mcp/my-server:latest | ssh your-linux-host &lt;span class="s2"&gt;"docker load"&lt;/span&gt;
ssh your-linux-host &lt;span class="s2"&gt;"sudo systemctl restart mcp-gateway.service"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;The key differences from macOS Docker Desktop:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concern&lt;/th&gt;
&lt;th&gt;macOS (Docker Desktop)&lt;/th&gt;
&lt;th&gt;Linux (headless)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;docker-mcp&lt;/code&gt; binary&lt;/td&gt;
&lt;td&gt;Bundled with Docker Desktop&lt;/td&gt;
&lt;td&gt;Downloaded from GitHub releases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;docker-pass&lt;/code&gt; plugin&lt;/td&gt;
&lt;td&gt;Proprietary binary&lt;/td&gt;
&lt;td&gt;Wrapper script → &lt;code&gt;docker-credential-pass&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secrets injection&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;docker mcp secret set&lt;/code&gt; + keychain&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;--secrets &amp;lt;env-file&amp;gt;&lt;/code&gt; (chmod 600)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Auto-start&lt;/td&gt;
&lt;td&gt;Docker Desktop&lt;/td&gt;
&lt;td&gt;systemd service&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Transport&lt;/td&gt;
&lt;td&gt;stdio or SSE&lt;/td&gt;
&lt;td&gt;SSE with &lt;code&gt;--transport sse&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mcp-remote&lt;/td&gt;
&lt;td&gt;Default settings work&lt;/td&gt;
&lt;td&gt;Needs &lt;code&gt;--allow-http --transport sse-only&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Everything else — catalog YAMLs, registry.yaml, the &lt;code&gt;docker mcp&lt;/code&gt; CLI — works identically between macOS and Linux once the above pieces are in place.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>linux</category>
      <category>mcp</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Correctly defining CDK dependencies in L3 constructs</title>
      <dc:creator>Daniel Schroeder</dc:creator>
      <pubDate>Mon, 28 Dec 2020 21:25:17 +0000</pubDate>
      <link>https://dev.to/aws-builders/correctly-defining-dependencies-in-l3-cdk-constructs-45p</link>
      <guid>https://dev.to/aws-builders/correctly-defining-dependencies-in-l3-cdk-constructs-45p</guid>
      <description>&lt;p&gt;When creating L3 constructs for the &lt;a href="https://aws.amazon.com/cdk/" rel="noopener noreferrer"&gt;AWS CDK&lt;/a&gt;, the easiest thing to get wrong is defining dependencies. And with wrong dependency definitions you make it hard to impossible to use your package. This post will show how to correctly define CDK dependencies.&lt;/p&gt;

&lt;p&gt;As a CDK user, you already know, all CDK core packages have to be of the same version or you will get cryptic errors such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Types of property 'node' are incompatible... Types have separate declarations of a private property 'host'.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So what you want to avoid when publishing L3 constructs, is to directly depend your package on any version of the core CDK packages. Still, this is the case in almost every L3 construct I have seen. I'm assuming this is because the core packages themselves do it like this and developers learn from reading code.&lt;/p&gt;

&lt;p&gt;Usually you find packages having either exact or caret versions in their &lt;code&gt;dependencies&lt;/code&gt;. Also, in most cases, these definitions are accompanied with the same items in the &lt;code&gt;peerDependencies&lt;/code&gt;. Let's analyze these two setups and what the result for the end-user is going to be:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dependencies with caret version&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^1.23.0"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"peerDependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^1.23.0"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The caret definition means, &lt;em&gt;install the latest minor compatible to the given version&lt;/em&gt;. That is everything &amp;lt; 2.0.0. So until CDK 2 drops, this will install the very latest package for &lt;code&gt;aws-lambda&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The end-user now can only install your package, when all application dependencies refer to the latest CDK packages as well. If CDK 1.70.0 or any other older version is used, there is no way the user can install your package without causing incompatibility between core packages. The solution from user perspective is to upgrade the CDK and deal with the potentially introduced breaking changes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dependencies with exact version&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.80.0"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"peerDependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.80.0"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A package with this &lt;code&gt;dependencies&lt;/code&gt; definition of course is only compatible with exactly one version of the CDK. This means your users cannot upgrade the CDK without upgrading your package and vice versa. To make your package usable by future CDK versions you need to release new versions of your package for every CDK release. Most probably you do this automated. And either you have a mapping between CDK version and your package version in your docs or you use the same exact version as the upstream packages, which renders the information (&lt;a href="https://semver.org" rel="noopener noreferrer"&gt;semver&lt;/a&gt;) of &lt;em&gt;your&lt;/em&gt; version string useless. How do you progress your own code? How do you communicate bug-fixes or breaking changes? And let's not forget, you waste computational resources for compiling, transferring and storing your build artifacts without actual change.&lt;/p&gt;

&lt;p&gt;The problem though, is not the format of your dependency definition (exact vs. caret) - the problem simply is: &lt;strong&gt;You have dependencies&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;And as simple as that problem statement, is the solution: &lt;strong&gt;Just don't&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fd5jxy1yo7mu87tzyylu9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fd5jxy1yo7mu87tzyylu9.png" alt="Thou shalt not list CDK core packages in thy dependencies" width="799" height="245"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Instead, list them in the &lt;code&gt;peerDependencies&lt;/code&gt; and &lt;code&gt;devDependencies&lt;/code&gt;. You should use caret versions, defining the minimum version &lt;strong&gt;required by your package&lt;/strong&gt;. Typically this should be the version when all your CDK dependencies went stable. If you don't know or don't care, use &lt;code&gt;^1.0.0&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"devDependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^1.0.0"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"peerDependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^1.0.0"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You need them in the &lt;code&gt;devDependencies&lt;/code&gt; to build your package with &lt;a href="https://aws.github.io/jsii/" rel="noopener noreferrer"&gt;jsii&lt;/a&gt; and you need them in the &lt;code&gt;peerDependencies&lt;/code&gt; to let the user know what packages need to be installed along with your package.&lt;/p&gt;

&lt;p&gt;Now, when a user installs your package, what happens depends on the used language and package manager. Either way, the user needs to define the peer-dependencies as dependencies of the application, e.g. in the &lt;code&gt;package.json&lt;/code&gt; or &lt;code&gt;requirements.txt&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;npm version &amp;lt; 6&lt;/strong&gt; will warn about missing peer dependencies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;npm WARN cdk-awesome-package@1.2.3 requires a peer of @aws-cdk/aws-lambda@^1.0.0 but none was installed.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same goes for &lt;strong&gt;yarn&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;warning " &amp;gt; cdk-awesome-package@1.2.3" has unmet peer dependency "@aws-cdk/aws-lambda@^1.0.0".
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In &lt;strong&gt;npm 6&lt;/strong&gt; for some reason this is missing and the user will only know about the missing dependency when the application is ran. That's fine though, the error message is clear about what package needs to be installed. An entry in the &lt;code&gt;package.json&lt;/code&gt; needs to be made:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.70.0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"cdk-awesome-package"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^1.2.3"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Starting with &lt;strong&gt;npm version 7&lt;/strong&gt; the handling of peer dependencies has changed - they will be automatically installed and you cannot override the version in the &lt;code&gt;dependencies&lt;/code&gt; section. Instead the user now needs to add it to the &lt;code&gt;overrides&lt;/code&gt; section.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"cdk-awesome-package"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^1.2.3"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"overrides"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@aws-cdk/aws-lambda"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.70.0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Alternatively the user can also just switch back to the old behaviour by setting &lt;code&gt;legacy-peer-deps=true&lt;/code&gt; in the &lt;code&gt;.npmrc&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;All other languages supported by jsii don't support peer dependencies. For these languages they are converted to normal dependencies. pip correctly interprets the caret version definition of &lt;code&gt;^1.0.0&lt;/code&gt; and in return you would install the latest version. For the dotnet package, jsii converted &lt;code&gt;^1.0.0&lt;/code&gt; to &lt;code&gt;1.0.0&lt;/code&gt;...  which in the end really doesn't matter. Because this can and has to be overridden by the user anyway.&lt;/p&gt;

&lt;p&gt;With &lt;strong&gt;pip&lt;/strong&gt; the user can override the version of dependencies by simply adding them to the applications &lt;code&gt;requirements.txt&lt;/code&gt; or &lt;code&gt;setup.py&lt;/code&gt;, e.g.:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;aws-cdk-aws-lambda==1.70.0
cdk-awesome-package&amp;gt;=1.2.3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In C# the user can also override the version in the project file, e.g.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight xml"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;ItemGroup&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;PackageReference&lt;/span&gt; &lt;span class="na"&gt;Include=&lt;/span&gt;&lt;span class="s"&gt;"Amazon.CDK.AWS.LAMBDA"&lt;/span&gt; &lt;span class="na"&gt;Version=&lt;/span&gt;&lt;span class="s"&gt;"1.70.0"&lt;/span&gt; &lt;span class="nt"&gt;/&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;PackageReference&lt;/span&gt; &lt;span class="na"&gt;Include=&lt;/span&gt;&lt;span class="s"&gt;"CDK.Awesome.Package"&lt;/span&gt; &lt;span class="na"&gt;Version=&lt;/span&gt;&lt;span class="s"&gt;"1.*"&lt;/span&gt; &lt;span class="nt"&gt;/&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/ItemGroup&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I haven't checked Java but I assume the same can be archived in a gradle file. &lt;/p&gt;

&lt;p&gt;And there you go. A CDK L3 construct working with any version of core CDK packages.&lt;/p&gt;




&lt;h4&gt;
  
  
  Update 2021/04/17
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;But what about breaking changes?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I've seen some developers are automatically releasing new versions of their L3 constructs for every new CDK release for the reason of compatibility, especially when experimental CDK components are involved. I strongly disagree with this approach.&lt;/p&gt;

&lt;p&gt;First of all, if an L3 uses experimental features, then the L3, as a consequence, is experimental. It should be used with caution and breaking changes should be expected with every minor update. Most likely it also should not be used in production, unless the user knows what she/he's doing and pays the proper attention. Using experimental features means, the user might not be able to upgrade without destroying/recreating related infrastructure.&lt;/p&gt;

&lt;p&gt;Next: Ensuring compatibility with future versions of a framework is not the responsibility of a package author. If there are breaking changes, of course things will fail. The same would be true if the user directly used experimental CDK features. The responsibility of catching these is not with the L3 author, but with the user of the construct. Every user needs to be aware that the smallest change needs testing. Of course, when there are package changes, the app needs to be deployed in a test environment, ran with &lt;code&gt;diff&lt;/code&gt; and ideally even has proper jest tests in place, before it reaches your prod environment.&lt;/p&gt;

&lt;p&gt;You cannot solve testing for the user. The best you can catch by building a package per CDK version is a change in the API signature. Maybe an option has been removed or renamed and therefore your code is not compatible and won't compile. The real danger though lies in functional changes, that make the code run, but behave differently. A very good example for this would be &lt;a href="https://github.com/aws/aws-cdk/releases/tag/v1.75.0" rel="noopener noreferrer"&gt;CDK version 1.75.0&lt;/a&gt;, which included the following change:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;efs: &lt;code&gt;keyId&lt;/code&gt; property uses the ARN instead of the &lt;code&gt;keyId&lt;/code&gt; to support cross-account encryption key usage. &lt;strong&gt;The filesystem will be replaced.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;strong&gt;This&lt;/strong&gt; is the real danger with experimental packages and you cannot protect the user from this by building a new version of your package.&lt;/p&gt;




&lt;p&gt;Also, what's exactly the benefit of building new packages for every release? You ensure it still builds. So what if it doesn't? Your build fails, you'll be notified by your action/workflow and you need to adjust the code to match the new API, because building new packages does not automagically fix the problem. Let's assume you're knee-deep involved in other projects or actual life and you don't have time to fix it for some time. &lt;br&gt;
&lt;strong&gt;Consequence: No user will be able to upgrade the core CDK packages until you fixed your code and released a compatible package.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Now imagine you had not auto-built new packages but instead used my suggested best practice? The package will not be compatible with the latest CDK version. &lt;br&gt;
&lt;strong&gt;Consequence: No user will be able to upgrade the core CDK packages until you fixed your code and released a compatible package.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The only difference would be, that you won't be notified by your failed pipeline/action and wouldn't be aware there is a breaking change. And this can easily be fixed by testing your package for new CDK releases.&lt;/p&gt;

&lt;p&gt;So instead of building and publishing new packages for absolutely no reason, you should &lt;strong&gt;test&lt;/strong&gt; your package against every new CDK release.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;What about &lt;a href="https://github.com/projen/projen" rel="noopener noreferrer"&gt;projen&lt;/a&gt;?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/projen/projen/pull/581" rel="noopener noreferrer"&gt;Since 1st of March 2021&lt;/a&gt; you can set &lt;code&gt;cdkDependenciesAsDeps: false&lt;/code&gt; in your &lt;code&gt;.projenrc.js&lt;/code&gt; to prevent projen from adding the CDK packages as &lt;code&gt;dependencies&lt;/code&gt; and only list them as &lt;code&gt;peerDependencies&lt;/code&gt;.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>cdk</category>
      <category>dependency</category>
    </item>
    <item>
      <title>Terraform workspace config organization</title>
      <dc:creator>Daniel Schroeder</dc:creator>
      <pubDate>Wed, 20 Nov 2019 15:23:55 +0000</pubDate>
      <link>https://dev.to/udondan/terraform-workspace-config-organization-42lh</link>
      <guid>https://dev.to/udondan/terraform-workspace-config-organization-42lh</guid>
      <description>&lt;p&gt;The suggested best practices for organizing configuration for multiple workspaces/environments is to call Terraform with &lt;code&gt;-var-file=$env&lt;/code&gt; to include a specific tfvars file.&lt;/p&gt;

&lt;p&gt;Of course this works. But it seems to be error prone if you allow to trigger an apply against any workspace with just any configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;terraform workspace &lt;span class="k"&gt;select &lt;/span&gt;production
terraform apply &lt;span class="nt"&gt;-var-file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;config/staging.tfvars
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Furthermore this cannot be used in Terraform Cloud, where you have to specify workspace related vars in the workspace configuration itself:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6ift1985ghsk5gmzchkb.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6ift1985ghsk5gmzchkb.png" alt="Terraform Cloud Variable configuration" width="800" height="497"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;There is no option for including tfvars per workspace.&lt;/p&gt;

&lt;h3&gt;
  
  
  Select config automatically based on the workspace
&lt;/h3&gt;

&lt;p&gt;Unfortunately there is no functionality to &lt;a href="https://github.com/hashicorp/terraform/issues/15966" rel="noopener noreferrer"&gt;automatically include a tfvars file based on the workspace name&lt;/a&gt; nor is there support for &lt;a href="https://github.com/hashicorp/terraform/issues/1478" rel="noopener noreferrer"&gt;conditionally including tfvars files&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;So you got to build something yourself. You can access the current workspace name via &lt;code&gt;terraform.workspace&lt;/code&gt;. There are a couple of things you can do with this value.&lt;/p&gt;

&lt;p&gt;Below are 3 solutions which all have the same exact outcome. The defined config is stored in &lt;code&gt;local.config&lt;/code&gt; and can be access via &lt;code&gt;local.config.ec2_instance_type&lt;/code&gt; etc.&lt;/p&gt;

&lt;p&gt;All 3 solutions support default values, so you're not required to define every config option in every environment.&lt;/p&gt;

&lt;p&gt;All examples are available in &lt;a href="https://github.com/udondan/example-terraform-workspace-config" rel="noopener noreferrer"&gt;this repository&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Inline expressions to select correct config from a map
&lt;/h3&gt;

&lt;p&gt;All config is held is a single file &lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/inline-locals/config.tf" rel="noopener noreferrer"&gt;&lt;code&gt;config.tf&lt;/code&gt;&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;locals&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;configs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

    &lt;span class="nx"&gt;_defaults&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;ec2_instance_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"t2.nano"&lt;/span&gt;
      &lt;span class="nx"&gt;regions&lt;/span&gt;           &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="nx"&gt;dev&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="c1"&gt;// use config from _defaults ^&lt;/span&gt;

    &lt;span class="nx"&gt;staging&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;ec2_instance_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"t2.medium"&lt;/span&gt;
      &lt;span class="nx"&gt;regions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"eu-central-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="nx"&gt;production&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;ec2_instance_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"t2.xlarge"&lt;/span&gt;
      &lt;span class="nx"&gt;regions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"us-west-2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"eu-central-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"ap-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nx"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;merge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nx"&gt;lookup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;configs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"_defaults"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nx"&gt;lookup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;configs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;terraform&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;workspace&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Straight forward&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cannot split config into separate files. Therefore could quickly get hard to maintain and compare environment config.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  2. Load config from YAML files
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://github.com/udondan/example-terraform-workspace-config/tree/master/include-per-yaml-file" rel="noopener noreferrer"&gt;Directory structure:&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;.
├── config
│   ├── _defaults.yml
│   ├── dev.yml
│   ├── production.yml
│   └── staging.yml
├── config.tf
├── main.tf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Example content of &lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-yaml-file/config/production.yml" rel="noopener noreferrer"&gt;&lt;code&gt;config/production.yml&lt;/code&gt;&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;ec2_instance_type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;t2.xlarge&lt;/span&gt;

&lt;span class="na"&gt;regions&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;us-east-1&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;us-west-2&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;eu-central-1&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;ap-east-1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The config is loaded in &lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-yaml-file/config.tf" rel="noopener noreferrer"&gt;&lt;code&gt;config.yml&lt;/code&gt;&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;data&lt;/span&gt; &lt;span class="s2"&gt;"local_file"&lt;/span&gt; &lt;span class="s2"&gt;"defaults"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;filename&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"${path.module}/config/_defaults.yml"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;data&lt;/span&gt; &lt;span class="s2"&gt;"local_file"&lt;/span&gt; &lt;span class="s2"&gt;"config"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;filename&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"${path.module}/config/${terraform.workspace}.yml"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;locals&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;merge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nx"&gt;yamldecode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;local_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;defaults&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;content&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nx"&gt;yamldecode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;local_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;config&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Straight forward&lt;/li&gt;
&lt;li&gt;Config for every environment resides in its own file&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No HCL expressions are possible in the config itself&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. Create a module per environment and return the config as an output
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://github.com/udondan/example-terraform-workspace-config/tree/master/include-per-module" rel="noopener noreferrer"&gt;Directory structure:&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;.
├── config
│   ├── _defaults
│   │   └── outputs.tf
│   ├── dev
│   │   └── outputs.tf
│   ├── main.tf
│   ├── production
│   │   └── outputs.tf
│   └── staging
│       └── outputs.tf
├── main.tf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In every &lt;code&gt;config/$env/outputs.tf&lt;/code&gt; a single output is defined like this:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-module/config/_defaults/outputs.tf" rel="noopener noreferrer"&gt;&lt;code&gt;config/_defaults/outputs.tf&lt;/code&gt;&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"data"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;ec2_instance_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"t2.nano"&lt;/span&gt;
    &lt;span class="nx"&gt;regions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-module/config/dev/outputs.tf" rel="noopener noreferrer"&gt;config/dev/outputs.tf&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"data"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="c1"&gt;// use config from _defaults&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-module/config/staging/outputs.tf" rel="noopener noreferrer"&gt;config/staging/outputs.tf&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"data"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;ec2_instance_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"t2.medium"&lt;/span&gt;
    &lt;span class="nx"&gt;regions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"eu-central-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-module/config/production/outputs.tf" rel="noopener noreferrer"&gt;config/production/outputs.tf&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"data"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;ec2_instance_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"t2.xlarge"&lt;/span&gt;
    &lt;span class="nx"&gt;regions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="s2"&gt;"us-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"us-west-2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"eu-central-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"ap-east-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Since you cannot use variables in a module &lt;code&gt;source&lt;/code&gt; parameter all 4 modules have to be defined in every environment. Furthermore you cannot directly access a module by name when the name is not hardcoded, so you need to additionally create a mapping like so:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-module/config/main.tf" rel="noopener noreferrer"&gt;config/main.tf&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;module&lt;/span&gt; &lt;span class="s2"&gt;"_defaults"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"./_defaults"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;module&lt;/span&gt; &lt;span class="s2"&gt;"dev"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"./dev"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;module&lt;/span&gt; &lt;span class="s2"&gt;"staging"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"./staging"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;module&lt;/span&gt; &lt;span class="s2"&gt;"production"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"./production"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;locals&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;data_map&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;dev&lt;/span&gt;        &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;dev&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;staging&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;staging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;production&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;production&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"data"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;merge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;_defaults&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;lookup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data_map&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;terraform&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;workspace&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the &lt;a href="https://github.com/udondan/example-terraform-workspace-config/blob/master/include-per-module/main.tf" rel="noopener noreferrer"&gt;&lt;code&gt;main.tf&lt;/code&gt;&lt;/a&gt; then the module needs to be loaded and for convenience the output gets registered as a local value:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="nx"&gt;module&lt;/span&gt; &lt;span class="s2"&gt;"config"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"./config"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;locals&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;config&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Config for every environment resides in its own file&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Complex setup&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Using modules as config provider seems to be the best solution, as you can split the configuration into separate files which supports HCL expressions. The setup though is complex and requires some additional boilerplate code for every additional environment.&lt;/p&gt;

&lt;p&gt;If you have no need for HCL expressions, the YAML solution seems to be nice as it is easy to setup and IMHO is very readable to humans.&lt;/p&gt;

</description>
      <category>terraform</category>
    </item>
  </channel>
</rss>
