Building FoxyInvoice — Chapter 7: CI/CD — push to main and it's live

This series is written in the open, from a real production system. This chapter is the deploy pipeline end to end, plus three real incidents where it quietly lied to us. [All chapters and diagrams live in the public repo.]

There is no staging environment in this system. There is main, and
main is production. That sounds reckless until you see the gates —
and the alternative for a solo operator (a staging box you maintain,
promote to "when there's time," and drift from reality) is worse.
This chapter is the pipeline that makes push-to-deploy safe enough to
sleep through, and the three times it lied to us anyway.

The pipeline, end to end

The deploy pipeline, rendered live: push → parallel gates (tests, security, secrets) → SSH deploy → health gates → SPA rebuild → smoke test → IndexNow.

Walk the interesting parts:

The gates run first, in parallel. Unit tests plus
Testcontainers-backed integration tests (real Postgres in ephemeral
Docker — tenant-isolation tests from Chapter 04 run here). A
conformance gate checks repo standards against a committed
baseline — new violations block, frozen historical debt doesn't, so
the fleet can adopt gates without a Big Bang cleanup. And gitleaks
scans for credentials, because everyone eventually pastes one.

A concurrency group serializes deploys. Two pushes minutes apart
deploy in order, never interleaved. (Keep this term in mind — it has a
war story below.)

The deploy is SSH + shell, nothing exotic. git reset --hard on
the host (why manual docker compose up there deploys stale code — the
host repo is always mid-flight), then a build that receives the private
feed token as a BuildKit secret — mounted during the build,
evaporating after; never in a layer or image history.

Health gates, then SPA, then smoke. Each API container must report
healthy before the pipeline proceeds (unhealthy = dump container logs
and fail loudly). Database migrations apply automatically on container
startup — EF Core tracks applied migrations in a history table, so
restarts are idempotent. Then the SPA rebuilds on the host (including
the SEO prerender step from Chapter 03), swaps into Caddy's directory,
and both /healthz endpoints must return 200. The final step even
pings IndexNow so discovery engines learn the content changed.

Runners: self-hosted, in a pool. GitHub's included minutes are
exhausted at a $0 spending limit, so jobs run on the org's own Linux
runners — shared with sibling repos, which means queueing is normal and
monitored. (This will matter in war story #3.)

The three times the pipeline lied

Every CI/CD system is a distributed system, and distributed systems
lie. Ours did, three ways, each now a permanent lesson:

1. The masked failure. The build step ended with
   || echo "WARNING: build failed (using cached image)" and then ran
   up -d --no-build. The build failed (missing token), the script
   shrugged, containers kept running yesterday's image — and the
   pipeline reported success for eight hours. The lesson, now a
   repo-wide rule: a failed build must abort the deploy. A green
   check that shipped nothing is worse than red.

2. The zombie concurrency holder. A cancelled deploy run never
   released the concurrency group. Every later run was created
   pending with zero jobs — the job row only appears when the lock is
   acquired — and sat there forever while the UI implied queuing was
   normal. Diagnosis came from the jobs API: zero jobs + idle runners =
   deadlock, not queue. The fix was renaming the group; the lesson is
   that "pending forever" is a distinct failure mode you must know how
   to recognize.

3. The label that matched nothing. Moving the deploy job to the
   self-hosted pool, the runner labels were written as a quoted
   string — which YAML/Actions parsed as one giant literal label
   no runner on Earth advertises. Jobs queued eternally again, this
   time invisible in a different way. The fix: a real YAML list. The
   general lesson: when nothing ever picks up your job, print exactly
   what labels it's demanding — don't trust your eyes reading YAML.

Rollback, and why we rarely say the word

Rollback here is git reset to the previous SHA and redeploy — minutes
of work, no image registry needed (everything builds on the host).
But database migrations complicate true rollback: a migration that
ran won't un-run safely. So the doctrine is forward-fix: the pipeline
is fast enough (and gates strong enough) that fixing forward beats
reverting schemas. The one hard rule that came from painful
experience: never apply schema changes out-of-band — hand-editing
the production DB twice caused startup crash-loops when the real
migration later collided with the hand-made change. If reality forces
your hand, the follow-up migration must be idempotent
(ADD COLUMN IF NOT EXISTS) and the history table reconciled.

Why push-to-deploy is a psychological feature

For a solo developer, the deepest value isn't the minutes saved — it's
that shipping stays a habit. When deploys are ceremony, you batch
changes, batches breed fear, fear breeds bigger batches. When git
push is the release process, every fix ships the day it's written
(and Chapter 10's feedback loop closes while the user still remembers
filing the report).

---

Recap. Gates before deploy, one deploy at a time, health-gated
containers, secrets that evaporate, smoke tests, and a philosophy of
forward-fix. The pipeline's job is to make shipping boring — and its
own failure modes taught us more than its successes.

---

Reading this and want to see a push go all the way through? Create a free workspace at
foxyinvoice.com, then redeem founding code
U8B4Z8S87X on the Upgrade page — 6 months of Pro, free, no card. If anything
breaks, there's a feedback button in the app. I read every one.

---

Next: Chapter 8 — Unit economics: what it actually costs.