Infrastructure and Deployment: Order by Iteration

Part of the "How to bring order to a project" series — the infrastructure track. Same starting point: a growing gamedev project, nearly two dozen services on different stacks — Python, Go, C++, a mobile client — and infrastructure that grew faster than it matured. Every service deployed differently, configs lived on servers, and a new service was born by copy-pasting the oldest one.

Attacking this with a "big refactoring" is a road to nowhere: the business won't let you stop shipping features for a month. What works is the iterative approach: small steps, each paying off immediately and breaking nothing.

Step 0. Snapshot and light

Two prerequisites, without which infrastructure changes turn into roulette:

• A state snapshot — the service map, who knows what, top risks. How to take one in a week — a separate article.
• Observability — logs, metrics, alerts. Also covered. The rule is simple: turn on the lights first, then move the furniture.

Now the actual infrastructure steps, in the order they pay off.

Step 1. CI/CD: templates instead of copy-paste

The classic disease: every service has its own CI config, written in its own era by its own person. You fix the pipeline in one repo — in the other twenty it stays broken.

The recipe that worked for me: a dedicated repository of CI templates. Inside:

• core-templates.yml — image builds, test runs, timings
• notify-templates.yml — notifications to the team messenger
• ready-made pipelines for the typical cases: a basic service, a service with secrets, a cluster deployment

A service plugs it in with one line:

include:
  - project: "infra/ci-templates"
    file: "/pipeline-basic.yml"

And gets the standard conveyor: on a merge request — test image build → tests → candidate build; on the main branch — the production image with a tag and latest. Everything goes through shared templates, so a fix or an improvement in one place rolls out to every service with its next pipeline.

Start with one service — the most frequently deployed one: the template gets polished there, the rest catch up as you touch them.

👉 The side effect matters more than the main one: deployment stops being one person's knowledge. Anyone can press the button in CI — and that's no longer scary, because before the button there are tests and a process that's identical for everyone.

Step 2. Configs into git, servers into code

The second source of chaos: configuration living directly on servers. Edited by hand, versioned nowhere, lost when a server moves.

There are two separate recipes here, and it pays not to mix them:

Static configs — what changes at deploy time: compose files, env templates, environment parameters. They belong in a configuration repository: change → merge request → it's clear who, what and why → rollout through CI.

Runtime configs — what changes the system's behavior on the fly: feature flags, prices, game parameters. These get their own GitOps repository with a schema and validation: values live in YAML, services read them through a small SDK. A product manager changes a parameter with a merge request, not with a ticket to the dev team — and the change history comes for free.

The servers themselves — only through Ansible roles (or any other IaC). The rule is hard: no hand edits on hosts. Not "just for a minute," not "I'll move it into the role later." Every manual edit is a future surprise during a migration or a recovery.

Step 3. Secrets: out of CI, into a vault

A separate pain: secrets smeared across CI variables, env files on servers and personal notes. The recipe I arrived at:

• Deploy a secrets manager (Vault, Infisical — the class of tool matters more than the brand) with test/production environments.
• CI variables keep exactly five values: the vault address, the machine-identity ID and secret, the project ID and the path. That's it.
• Everything else — registry credentials, deploy SSH keys, notification webhooks — the pipeline fetches from the vault itself, per environment.

A technical detail that saved hours of debugging: every secret is written to its own file via base64 -d, byte for byte, with no shell interpretation. Multiline SSH keys and certificates stop breaking on escaping once and for all.

The win isn't only security: rotating a secret is an edit in one place, not archaeology across twenty repositories. And when someone leaves the team, you know exactly what to reissue and where.

Step 4. Deployment: one entrance, different backends

Deployment in the CI templates is standard too, with two backends for different scales:

• Simple services live on hosts under docker compose: the deploy job takes the host list and the SSH key from the vault, connects and updates the service. Boring and predictable.
• Cluster services ship to Kubernetes via Helm: the repo holds values.yaml + values.production.yaml, the image tag comes from the commit SHA, the job waits for the rollout and logs its progress. Merge requests build a candidate with its own tag — deployable to a test environment with the very same template.

Both roads end the same way: a notification in the team messenger — who deployed what and where. It's cheaper than any "release book": the deploy history assembles itself in the channel.

👉 The general principle: a service has no deployment method of its own. It has parameters (hosts or helm values) — the method is shared. A new deployment method appears in the templates, or it doesn't appear at all.

Step 5. Backups: order isn't there until you can restore

A test that puts a lot into perspective: "the server is dead for good — how many hours until everything runs again?" If the answer starts with "well…" — the track has further to go.

• Database backups — on schedule, to a separate location, with an alert on "backup didn't happen"
• Recovery is tested by hand at least once a quarter: a backup that has never been restored is a lottery ticket, not a backup
• Everything else must be restorable from git: roles bring up the server, CI builds and ships the services, configs arrive from repositories

When this genuinely works, "moving a server" turns from a special operation with a midnight call into a half-day procedure.

Step 6. The reference service template

Once CI is standardized and configs are in git, you can finally pin down "what a proper service looks like." My reference consists of:

• Dockerfile — multi-stage, with separate targets for tests and production (the CI templates are built for exactly that)
• Repo structure — code, tests, migrations, a README with a two-command local start
• JSON logs with a request_id passed through — so the service plugs into centralized logging right away
• A healthcheck endpoint — so the orchestrator and monitoring see liveness without magic
• The CI templates plugged in — a pipeline out of the box
• A service passport — one page: why it exists, its area of responsibility, boundaries, who owns it, how it deploys

The passport is the underrated part. Passports add up to a living map of the system: a service registry with ports, stacks and owners, plus a "who calls whom" graph. With twenty services, that map answers half of the architecture questions before they're asked.

A new service is created from the template in minutes. Old ones are not rewritten en masse: the rule "every touch brings the service closer to the reference" pulls everything alive up to the standard within months — and what's dead gets to die honestly.

What came out of it

• Deploying any service is one button in CI, the same process for everyone
• Configs and servers are restorable from git — "moving a server" went from special operation to procedure
• A new service comes from the template in minutes, with logs, metrics and a pipeline from day one
• Secrets rotate in one place; someone leaving the team is a procedure, not a panic
• The deploy history assembles itself in the notification channel — who shipped what and where
• Infrastructure knowledge stopped living in one head: passports + templates + roles are readable by any team member

The track's checklist

⬜ CI templates in a dedicated repo; the first service plugged in

⬜ Tests are mandatory on merge requests; no deploys bypass CI

⬜ Static configs — in a configuration repository

⬜ Runtime parameters — a GitOps repo with a schema and an SDK

⬜ Servers — only through IaC roles; no hand edits on hosts

⬜ Secrets — in a vault; CI keeps only the connection parameters

⬜ Deployment — shared, from templates (SSH-compose / Helm), with a chat notification

⬜ Database backups on schedule + a quarterly restore test

⬜ Service reference: Dockerfile, structure, JSON logs, healthcheck, passport

⬜ Rule: every touch brings an old service closer to the reference

The next tracks of the series — architecture, people, processes — are on the map.

---

Originally published at mind.mansur.expert.