Securing a CI/CD Pipeline Behind NAT with Tailscale

A Spring Boot REST API running on a home server behind NAT, deployed via GitHub Actions through a self-hosted runner installed on the same production machine. It worked for months — until a security audit flagged it as critical severity. The Docker socket on that runner meant any workflow step could own the server. A fork PR on a public repo could execute arbitrary code on production.

This article covers two problems solved simultaneously: how to harden a CI/CD pipeline on a public repository, and how to deploy to a server with no public IP from ephemeral GitHub-hosted runners. The migration took ~8 hours across 2 days, with a 2-week observation period before decommissioning the old runner.

Why a self-hosted runner on a public repo is a ticking bomb

The project is a solo-maintained API serving production traffic through Cloudflare tunnels. The CI/CD pipeline ran on a self-hosted GitHub Actions runner installed on the same machine that hosted the application. For months, this was convenient: no network hop for deploys, direct Docker socket access, zero latency.

Then I read the Shai-Hulud writeup and realized what I had built: a remote code execution endpoint on my production server, accessible to anyone who could open a pull request.

Here's the risk matrix the audit produced:

Risk	Impact
Docker socket = root on host	docker run --privileged -v /:/hostfs alpine chroot /hostfs gives full root access
Persistence between jobs	A malicious workflow can install cron jobs, modify ~/.bashrc, or alter the runner binary
No network isolation	Runner sits on the home LAN with no VLAN separation — lateral movement to other devices
Public repo + self-hosted runner	Any fork can open a PR and execute code on the runner — zero-click RCE for external attackers
Secrets persisted on disk	.env files and JWT keys remain after deploy — credentials exposed if server is compromised

The CVE-2025-32955 disclosure — showing that disable-sudo in Harden-Runner could be bypassed via Docker — confirmed this wasn't theoretical.

The fix was clear: move to GitHub-hosted runners (ephemeral VMs, destroyed after each job, no Docker socket exposure). But my server had no public IP.

The NAT problem: how do you SSH into a server with no public IP?

The production server sits behind a residential NAT with no port forwarding and no static IP. GitHub-hosted runners spin up in Azure — they can't SSH into 192.168.x.x.

I tried two approaches before finding one that worked in automation.

Cloudflare Access SSH: the wrong tool for CI/CD

I first tried Cloudflare Zero Trust with Service Tokens. The idea: create an HTTP tunnel (cloudflared) on the server, authenticate the runner via Service Token, and proxy SSH over HTTPS.

It worked in manual testing. It failed in CI/CD. Multiple workflow runs produced:

error: username is empty
error: websocket: bad handshake
error: in libcrypto

The root cause: Cloudflare's Service Token authentication doesn't reliably forward the SSH username from the runner to the server. Behavior varied between cloudflared versions — a known bug in v2026.6.0 made it worse. After two days of debugging, I scrapped the approach.

Tailscale WireGuard: the solution that actually works

Tailscale creates a WireGuard mesh network. The key insight: both the server and the runner become nodes in the same encrypted mesh, regardless of NAT.

The setup:

1. Server: curl -fsSL https://tailscale.com/install.sh | sh && tailscale up --ssh — joins the mesh with a stable IP (100.x.x.x) and enables Tailscale SSH (OIDC-based auth, no SSH keys needed).
2. Runner: the tailscale/github-action creates an ephemeral node in the mesh using an auth key. The node is destroyed when the job ends.
3. Connection: ssh deploy-user@100.x.x.x — direct, encrypted, no intermediate proxy.

Why Tailscale won over Cloudflare for CI/CD:

Aspect	Cloudflare Access SSH	Tailscale
Auth mechanism	Service Token (HTTP headers)	OIDC via auth key (WireGuard)
Reliability in automation	Intermittent failures	Consistent
Binary required on runner	cloudflared (download each job)	None (GitHub Action handles it)
Transport	HTTPS (HTTP tunnel)	WireGuard (UDP, end-to-end encrypted)
Cost	Free (Zero Trust Free tier)	Free (Personal plan, 1,000 ephemeral min/month)

7 layers of defense: from SHA pins to rollback automation

Moving to hosted runners eliminated the Docker socket and persistence risks. But a hardened pipeline needs more. Here are the 7 layers I implemented, each one addressing a specific attack vector.

Layer 1: SHA-pinned actions

After the tj-actions/changed-files supply chain attack (March 2025), I pinned every action by its full commit SHA — not by mutable tag.

# Vulnerable to tag hijack
- uses: actions/checkout@v4

# Pinned to immutable commit SHA
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

I maintain a verification script that audits all SHAs against the GitHub API before each release. Dependabot PRs for github-actions ecosystem keep them updated weekly.

Layer 2: minimal permissions

Every workflow starts with the tightest possible scope:

permissions:
  contents: read  # workflow-level default

jobs:
  build:
    permissions:
      contents: read
      packages: write  # only this job pushes to GHCR
  deploy:
    permissions:
      contents: read
      packages: read   # only pulls from GHCR

No id-token: write, no pages: write, no broad write-all. The GITHUB_TOKEN is scoped per-job and discarded when the ephemeral VM is destroyed.

Layer 3: script injection prevention

GitHub Actions expressions like ${{ github.event.inputs.ref }} are string-interpolated before the shell runs. A malicious input can inject arbitrary commands:

# Vulnerable: user-controlled value directly in shell
- run: echo "Deploying ${{ github.event.inputs.ref }}"

# Safe: pass through env var (shell-quoted by the runner)
- env:
    INPUT_REF: ${{ github.event.inputs.ref }}
  run: echo "Deploying ${INPUT_REF}"

I audited every run: block and moved all user-controllable expressions (github.event.inputs.*, github.ref_name, github.actor) to env: blocks. Three steps in the deploy workflow were fixed.

Layer 4: Harden-Runner in audit-then-block mode

step-security/harden-runner runs as the first step in every job:

- name: Harden Runner
  uses: step-security/harden-runner@002fdce3c6a235733a90a27c80493a3241e56863 # v2.12.1
  with:
    egress-policy: audit

Start with audit mode. Harden-Runner monitors all network connections, file access, and process execution without blocking anything. Let it run for 2-4 weeks across real deployments to build a baseline of legitimate endpoints (GHCR, Maven Central, Tailscale coordination servers, etc.).

Once the baseline is stable, review the StepSecurity dashboard and migrate to block mode with an explicit allowlist:

- name: Harden Runner
  uses: step-security/harden-runner@002fdce3c6a235733a90a27c80493a3241e56863 # v2.12.1
  with:
    egress-policy: block
    allowed-endpoints: >
      ghcr.io:443
      github.com:443
      plugins.gradle.org:443
      repo.maven.apache.org:443

This two-phase approach avoids false positives breaking deploys on day one while giving you a concrete path to enforcement.

Layers 5-7: deploy scripts, GHCR auth, and rollback

The remaining three layers handle the deploy mechanics:

Layer 5 — External deploy scripts. Deploy logic was extracted from inline YAML heredocs into versioned shell scripts (.github/scripts/):

.github/scripts/api/
├── deploy.sh      # pull, retag, restart, healthcheck
├── rollback.sh    # restore :backup image
├── verify.sh      # pgbackrest + container status
└── cleanup.sh     # docker image prune

Scripts are copied to the server via scp and executed via ssh. This eliminates fragile YAML heredocs, complex quoting, and sed -i strip hacks. Each script has a trap for cleanup on exit.

Layer 6 — GHCR auth without exposing tokens. Instead of exporting GITHUB_TOKEN as an environment variable over SSH (which would appear in process lists), we use Docker's config.json: the runner creates ~/.docker/config.json with valid GHCR credentials, scp copies it to the server, and deploy.sh runs docker --config /path/to/config pull ghcr.io/... — the token is read from file, never in an env var on the remote host.

Layer 7 — Concurrency, timeouts, and rollback. Every job has a timeout-minutes (build: 15, deploy: 25, verify: 15, cleanup: 5). The deploy script creates a :backup tag of the current image before pulling the new one. If the health check fails (30 attempts × 5s = 150s), the rollback step restores the backup automatically.

concurrency:
  group: deploy-api-prod
  cancel-in-progress: false  # never cancel a running production deploy

The final architecture: 4 jobs, each on its own ephemeral VM

The production deploy workflow (deploy-api-prod.yml) orchestrates 4 jobs:

1. Build & Push Image — ubuntu-24.04, 15min timeout, Gradle build → Buildx → GHCR, packages: write
2. Production Gate — Manual approval required
3. Deploy to Production — ubuntu-24.04, 25min timeout, Tailscale → scp scripts → ssh deploy.sh, packages: read
4. Verify Deployment — ubuntu-24.04, 15min timeout, pgbackrest check + container status
5. Cleanup — ubuntu-24.04, 5min timeout, docker image prune, if: always()

Key design decisions:

• Image tagging: every push creates 3 tags in GHCR — <git-tag>, sha-<full-commit> (immutable), and latest.
• Pull by SHA: the deploy script pulls by sha-<commit> (immutable) and retags to the local compose image name. The docker-compose.yml is never modified.
• Tailscale in every job: each job (deploy, verify, cleanup) sets up its own ephemeral Tailscale node. Jobs run on separate VMs — there's no shared state.

Here's the core of the deploy job — the part that connects to the server:

deploy:
  runs-on: ubuntu-24.04
  needs: build
  environment:
    name: production
  permissions:
    contents: read
    packages: read

  steps:
    - name: Setup Tailscale
      uses: tailscale/github-action@306e68a...  # v4.1.2
      with:
        authkey: ${{ secrets.TS_AUTH_KEY }}
        tags: tag:ci

    - name: Login to GHCR
      uses: docker/login-action@0d4c9c5...  # v3.2.0
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Deploy via SSH
      uses: ./.github/actions/api/deploy-via-ssh-api
      with:
        ssh-target: ${{ env.SSH_TARGET }}

The composite action (deploy-via-ssh-api) handles scp of secrets, scripts, and Docker config — keeping the workflow YAML clean and the logic testable.

Before and after: the numbers

Metric	Before	After
Attack surface	Docker socket + home LAN + persistent runner	SSH over WireGuard to ephemeral VMs
Secrets on disk during deploy	.env + JWT keys persisted indefinitely	Injected via SSH, cleaned up by trap on exit
CI/CD cost	$0 (approaching usage-based pricing)	$0 (public repo, within free tier)
Deploy time (cold start)	~8-12 min	~6m 38s
Deploy time (cached)	—	~3m 29s
Supply chain protection	Tags (mutable)	SHA pins (immutable) + Dependabot
Manual steps to deploy	Push to branch → auto-deploy	Push tag → auto-build → manual approval → deploy

The deploy cold start dropped from 8-12 minutes to 6m 38s. Cached deploys hit 3m 29s. The CI/CD cost stayed at $0 — the public repo is within GitHub Actions' free tier, and Tailscale's Personal plan covers the ephemeral minutes.

What I'd do differently

Cloudflare Access SSH is not reliable for CI/CD automation. It works perfectly for manual access, but Service Token authentication behaves inconsistently across cloudflared versions. If your server is behind NAT and you need automated SSH, Tailscale (or any WireGuard mesh) is a more reliable choice. I wasted two days debugging Cloudflare before switching.

StrictHostKeyChecking=no is acceptable inside a WireGuard mesh. I initially spent time trying to dynamically extract host keys from Tailscale. I ultimately decided it wasn't worth the complexity: WireGuard already provides end-to-end encryption and node identity verification. The risk acceptance was documented internally — WireGuard's encryption model made SSH host key verification redundant.

Extract deploy logic from YAML early. My first iteration had 50-line heredocs inside run: blocks with complex quoting, sed -i to strip indentation, and nested command substitutions. Moving to external .sh scripts eliminated an entire class of bugs and made the deploy logic testable in isolation.

GHCR auth via config.json beats environment variables. Exporting GITHUB_TOKEN over SSH puts it in the remote process list (/proc/*/environ). Copying Docker's config.json via scp and using docker --config keeps the token out of the process tree entirely.

The full write-up went through 34 revision cycles. Most of those were me discovering that the "simple" version of each layer had edge cases I hadn't considered — like the script injection fix requiring an audit of every run: block, not just the ones I thought were user-controlled.

What's next

The workflows and deploy scripts are available in the public repository: unspoken-tech-org/workshop_rest_api.

The runner migration was the foundation. The real hardening came from the layers on top: SHA-pinned actions, minimal permissions, script injection prevention, external deploy scripts, and secure token handling. Each layer is simple individually; together, they create meaningful defense-in-depth — even (or especially) for a solo developer maintaining a public repository.

Have you hit the self-hosted runner problem on a public repo? I'm curious whether anyone has found a cleaner way to handle the NAT traversal — or if Tailscale's ephemeral node pattern is as good as it gets for this use case.