Inside twio's CI/CD: How a Two-Engineer AI Startup Ships

No per-push CI, exactly one hard gate, and LLM evals that never block a merge — and why.

twio is an AI workspace for mortgage advisers. Its core features — email parsing, loan refixing, application-pack preparation — are all LLM-driven. The engineering team is two people. There is no dedicated QA and no dedicated ops. The stack is a Node/TypeScript monolith with a React frontend, running on GCP Cloud Run, backed by Neon Postgres.

That combination of constraints — a tiny team, an LLM-centric product, fully managed infrastructure — produced a CI/CD pipeline that looks noticeably different from the textbook version: there is no "run CI on every push," the entire pipeline has exactly one mandatory gate, and LLM evals never block anything. This post walks through the whole flow along two tracks — CI (how code gets verified) and CD (how code reaches an environment) — and explains the reasoning behind each design decision.

The big picture

The pipeline consists of seven GitHub Actions workflows, one Cloud Build trigger on the GCP side, and one git hook that has since been deleted:

Part 1: CI — how code gets verified

CI has to answer two questions: what verification exists, and when each kind runs.

1.1 Four layers of verification

The first two layers are unremarkable. The third and fourth are specific to LLM products and worth unpacking.

LLM evals. Each eval is a Vitest test: it assembles the real system prompt for the unit under test (a tool, a sub-agent, or a workflow step), calls a real model, but mocks every tool return — no database, no network beyond the LLM. Cheap deterministic assertions run first, then a Gemini-based LLM-as-judge checks the semantic claims. "Real model" implies two properties: it costs money, and any single run is noisy — the same code can pass today and fail tomorrow. Those two properties dictate where evals can live in CI: on-demand or as monitoring, never as a gate. Our working agreements (the CLAUDE.md at the repo root) encode this as a hard rule: "Gate only on stable, discriminating signals."

End-to-end. The opposite extreme: real database, real model, real /api/chat, driving an entire business flow as a black box — for example, a loan refix from creating the contact all the way to sending the application to the lender. A cold-start case takes minutes. It runs locally, by hand, and never appears in any workflow.

1.2 The rise and retirement of the pre-push hook

The original trigger mechanism was a Husky pre-push hook. The entire thing was one line:

# .husky/pre-push
npm run build && npm test

Build plus the full unit suite, enforced locally before every push. As the test suite grew, this model hit its limit: roughly five minutes of waiting per push, on a high-frequency action. Waiting cost multiplied by push frequency stopped being worth it. In late April the hook was deleted and verification moved into CI, on demand. The commit message is refreshingly plain:

Move build-and-test validation from local Husky pre-push into a comment-triggered GitHub Action so contributors can run checks consistently in CI.

Beyond the waiting, consistency was the second motive: a hook only constrains machines that have it installed, local environments drift, and git push --no-verify walks straight around it. A check that runs in CI is identical for everyone — and its result is recorded on the PR.

1.3 On-demand verification, driven by PR comments

The pre-push hook was replaced by three PR comment commands:

The architecture is "one router plus N reusable workflows": a single workflow, pr-comment.yml, listens for issue_comment events, parses the command with a regex, and invokes the matching workflow via workflow_call. Adding a new command means one new regex branch in the router and one new workflow file — the trigger logic never changes, and you never end up with several workflows racing for the same event.

A few deliberate choices here:

• On demand, not on every PR. The full unit suite costs five minutes; evals cost real API money; and plenty of PRs never touch LLM-adjacent code. The author decides what to run. Single-file mode (say, /eval:tool build_refix_proposal_from_text) drives the cost of one verification about as low as it can go.
• A closed interaction loop. A recognized command immediately gets a 🚀 reaction ("heard you"), and the run posts its result — ✅/❌, commit, command, log link — back onto the PR. A new command on the same PR cancels the previous run (the concurrency group is keyed by PR number).
• Fork safety. Evals need LLM API keys, which live in a GitHub Environment called LLM Test. Workflows triggered from fork PRs can't see those secrets, so the router detects forks and replies with a polite refusal instead. /ut needs no keys and runs on forks just fine.

1.4 After the merge: nightly-eval as the safety net

On-demand verification has a structural hole: it relies on discipline, and sooner or later a PR merges without enough eval coverage. The plug for that hole is nightly-eval — the full eval:tool and eval:subagent suites against main, every night:

on:
  schedule:
    - cron: "17 15 * * *"   # ~3 a.m. NZ; deliberately off the top
                            # of the hour, the slot GitHub delays most
  workflow_dispatch: {}      # can also be triggered manually
concurrency:
  group: nightly-eval
  cancel-in-progress: false  # never cancel an in-flight eval run

Its most important property is that it is report-only: it observes, it never blocks. The workflow's header comment gives two reasons:

1. These evals call real models and grade with an LLM judge, so a single pass/fail is noisy — in the comment's words, "unfit to gate merges on."
2. It runs after the merge. The code is already on main; failing can't stop anything. Its entire value is trend monitoring.

The alerting channel is self-healing: a failure opens an issue labeled eval-regression; consecutive failures append comments to that same issue rather than opening a new one every day; a green run posts "Recovered" and closes it. At any moment there is at most one open regression issue, which makes the Issues tab a de facto health light for LLM behavior on staging.

1.5 The only hard gate: before production

The whole pipeline has exactly one mandatory checkpoint, and it lives inside the production deploy workflow (deploy-prod.yml):

jobs:
  test:                        # the only hard gate
    steps:
      # ...checkout main, npm ci...
      - run: npx tsc --noEmit
      - run: npm test
  deploy:
    needs: test                # no green, no deploy

Why only one gate, and why only deterministic checks? A gate's job is to stop things that shouldn't ship, which means it has to stand on stable signals. A gate that goes red at random gets ignored or re-run until it passes — and takes the pipeline's credibility down with it. So type checks and unit tests gate the release; LLM quality is the nightly monitor's job.

Part 2: CD — how code reaches an environment

2.1 The environment model

Two environments, fully symmetrical, in the same GCP project and region:

The branch model: all day-to-day work lands on main through PRs. The prod branch takes no direct commits — it is rebased onto main at each release. Every position of prod therefore corresponds to an actual production release: it is both the deploy source and the release history.

2.2 Staging: every push is a deploy

Code merged into main shows up on dev.twio.ai a few minutes later. GitHub Actions does nothing on this path — a Cloud Build trigger on the GCP side watches the repository, builds from the Dockerfile at the repo root, and deploys to twio-main.

Why we're comfortable with a zero-gate staging: mistakes are cheap (it's an internal trial environment), safety nets exist (nightly evals surface an LLM regression within a day at worst, and the release gate stops deterministic breakage from reaching production), and the fix path is short (the next push is the next deploy). Flip it around: a gate here would tax every single push with a wait, and buy protection the release gate already provides.

Worth stating honestly: after the pre-push hook was deleted, this became the most aggressive trade-off in the whole design. A PR that never ran /ut can, in principle, put code that fails unit tests onto staging — until someone runs the tests, the nightly alarm fires, or the release gate catches it. For a two-person team that risk is acceptable. For a bigger team it wouldn't be.

2.3 Production: a manually triggered pipeline

A release is one click of "Run workflow" in the GitHub Actions tab. Five steps follow:

The reasoning, step by step:

• Manual, not automatic. Release cadence is a human decision: batch a few PRs, avoid customers' active hours. A small team doesn't need merge-to-production continuous deployment; it needs "one click when we choose to ship, verification guaranteed every time."
• Rebase, not merge. prod stays commit-for-commit identical to main (linear history), and each release leaves an exact snapshot. The push uses --force-with-lease rather than bare --force: if the remote branch moved unexpectedly, the push fails instead of silently overwriting. If prod doesn't exist, it's created from main — a defense added later covering the deleted-or-never-created case.
• Keyless auth via WIF. GitHub authenticates to GCP through Workload Identity Federation: no service-account JSON key stored anywhere, GitHub's OIDC token is exchanged for short-lived credentials, and the trust relationship is scoped to this one repository. A key that doesn't exist can't leak.
• Pinning APP_BASE_URL. This variable is boot-critical — the container uses it to provision its task queues at startup, and without it the service won't come up (the workflow comment: "its absence crashes boot"). So every deploy re-asserts it via --update-env-vars, protecting against someone editing service config in the console, losing the variable, and getting a mysterious explosion several deploys later.
• A health check as the last word. A "successful" Cloud Run deploy only means the new revision started; it doesn't mean the app is healthy. After deploying, the workflow curls the service URL and fails the run on anything outside 200–399.
• Uncancelable. The concurrency group sets cancel-in-progress: false — in the comment's words, "never cancel a prod deploy mid-flight." New triggers queue up; a half-finished production deploy never gets killed. (The old staging workflow was configured the opposite way, cancel-in-progress: true: a newer push simply supersedes the deploy in progress, because staging only ever cares about the latest version.)

2.4 Self-contained at runtime: no post-deploy checklist

The container's start command:

CMD ["sh", "-c", "node dist/db/migrate.js && node dist/index.js"]

Database migrations run automatically on every container start. The app then bootstraps all of its infrastructure dependencies before binding the port — creating or reconciling Cloud Tasks queues, Cloud Scheduler jobs, and Pub/Sub topics. If critical configuration is missing, startup fails loudly (the container won't come up and the deploy goes red) instead of limping along.

For a small team, the point of this choice is that deploys have no runbook. There is no "remember to run migrations after deploying," no "remember to create the queue" folklore — environmental dependencies are either handled by the code itself or the deploy fails and tells you immediately. Health check + loud failure + self-provisioning together guarantee that a green deploy means the environment is correct.

2.5 Cleanup: cost is part of the pipeline

Every gcloud run deploy --source produces a new container image (stored in Artifact Registry, billed by storage) and a new Cloud Run revision. Two environments deploying at high frequency means the bill only goes one way unless something pushes back. The hardening commit recorded the motivation:

Each deploy via gcloud run deploy --source . accumulates Docker images in Artifact Registry, driving up storage costs.

cleanup-images.yml does the housekeeping, hourly (plus on every push, plus manual):

• Cloud Run revisions: each service keeps only the newest one plus whatever is serving traffic;
• Images: each service keeps the two most recent; anything referenced by a live revision is skipped; the rest are deleted.

Its own history tracks usage growth: it began as an inline cleanup step in the deploy workflow, was split into a standalone daily cron the same day, and moved to hourly a month later once deploy frequency picked up.

Summary

Every trigger point in the pipeline:

Each decision is small on its own. Together they reduce to three consistent principles:

1. Gate only on deterministic signals. The pipeline's single mandatory checkpoint (before production) contains nothing but type checks and unit tests. Making a randomly-red check a gate just teaches everyone to ignore gates.
2. Noisy signals monitor; they don't block. The value of LLM evals is in the trend: run nightly, open an issue on failure, auto-close on recovery — not in vetoing any particular merge.
3. Expensive operations run on demand. The time-expensive (five minutes of unit tests) and the money-expensive (real-model evals) are both behind explicit commands, with the author deciding what to run; whatever slips through is caught by the nightly net and the release gate.

Finally, the fine print: this design assumes a team small enough for mutual trust, a staging environment where mistakes are cheap, and a release cadence of two or three ships a week. As the team grows, as compliance requirements appear, or as evals get stable enough to trust, the number and placement of gates should be revisited. This pipeline is itself the product of several rewrites within a single year — there's no reason to believe today's shape is final.

---

Originally published on Medium.