One path to production for every service. The pipelines don't know which service they're shipping - they read that off the manifest. What's left to a person is three decisions, and one check that a version string is really in the binary.
👋 I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. Earlier parts of this series were about the manifest that declares a service and about what you get the moment you declare it. This part is about the last thing that falls out of that same file: how the code actually reaches an environment. Notes: github.com/brilliant-almazov.
The thought I want to share is narrow: deployment stopped being a per-service artefact for us, and the reason it could stop is that there's exactly one place that says what a service is made of. Maybe you've already got this and did it better. Maybe you looked at it and decided a per-service pipeline is worth the duplication. I'd genuinely like to hear which.
The defect that stays green
Start with the failure, because it's the one that shaped everything else here.
Go stamps build metadata into a binary with linker flags:
-ldflags "-X example.com/platform/lib/go/platform.Version=$(VERSION) \
-X example.com/platform/lib/go/platform.Commit=$(COMMIT)"
Here's the part that costs money: the linker silently ignores -X pointed at a symbol that doesn't exist. No warning. No non-zero exit. The build is green, the image is pushed, the tag looks right in the registry, and the variable inside the binary still holds its default - dev.
-X points at a symbol that does not exist
│
▼
build is green — no warning, no non-zero exit
│
├──▶ version = dev in the audit trail
└──▶ version = dev in the trace attribute
The error is invisible everywhere except the version field.
And dev doesn't sit still. It rides into the places that consume the version:
- the audit trail, where every record now claims it was written by
dev; - the version attribute on the trace, so the spans of three different releases are indistinguishable.
Nothing breaks. Nothing alerts. The error is invisible everywhere except the version field - and you find it on the day you needed the version to find something. Which is, by construction, the worst day.
There were two aggravating factors in our case:
-
The template historically carried a dead target. Every service generated from it inherited an
-Xaimed at a symbol that isn't there. So this wasn't one service's mistake - it was the default. -
There are two places to fix, and fixing one is worse than fixing neither. The flags live in the
Makefileand in theDockerfile. Fix theMakefileonly, and a local build stamps correctly while the image - the thing that actually runs - doesn't. Now local and production disagree, and local is the one you'd check.
The flags target the platform's symbol, and they get fixed in both files, together, or the fix isn't a fix.
A check, not more care
The obvious response to "the template had a dead target" is a rule: always point ldflags at the platform symbol, always in both files. I've written rules like that. They work until the person writing the next service hasn't read them, which is always.
What works is a check. There's a build target that:
- builds both binaries with a deliberately fake version string;
- greps that string out of the produced binaries;
- fails if it isn't there.
The shape of it is unremarkable:
verify-version:
go build -ldflags "-X <platform-symbol>.Version=vX.Y.Z-probe" -o bin/server ./cmd/server
strings bin/server | grep vX.Y.Z-probe
Empty output means the symbol is wrong. That's the whole diagnostic. It doesn't matter whether the symbol is wrong because someone renamed a package, or because the template was wrong, or because a copy-paste dropped a path segment - the probe string is either in the bytes or it isn't.
This is the same move as everywhere else in this block of the series: when something can be wrong silently, the answer isn't to be more careful, it's to make the wrongness produce output. A rule you can violate while the build stays green is not a control.
build both binaries with a deliberately fake version string
│
▼
strings bin/server | grep vX.Y.Z-probe
│
├──▶ the string is found ──▶ the symbol exists
└──▶ empty output ──▶ the symbol is wrong
A rule you can violate while the build stays green
is not a control.
Three pipelines, identical for every service
With that out of the way, here's the whole delivery surface. Three pipelines, the same for every service in the fleet.
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ build │ │ tests │ │ deploy │
│ on tag vX.Y.Z │ │ on PR · on push · │ │ manual │
│ │ │ nightly │ │ │
│ one Dockerfile, │ │ │ │ two parameters: │
│ one build-arg per │ │ 1 build │ │ │
│ binary │ │ 2 linter │ │ version │
│ │ │ 3 containers up │ │ server / worker / │
│ images to the │ │ (one script) │ │ all │
│ registry under the │ │ 4 run with coverage │ │ │
│ version tag │ │ 5 time budget │ │ then one step, the │
│ │ │ 6 teardown │ │ same step for every │
│ │ │ if: always() │ │ service │
│ │ │ 7 coverage report │ │ │
│ │ │ 8 threshold check │ │ │
│ │ │ 9 ratchet │ │ │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
1. Build - on a tag
Triggered by pushing a tag vX.Y.Z. It's a matrix job over the binaries: each one comes out of the same Dockerfile, parameterised by which binary to build:
docker build --build-arg BINARY=server .
docker build --build-arg BINARY=worker .
Images go to the registry under the version tag. Version and commit are stamped with the ldflags above - which is why the check in the previous section exists at all.
Note what isn't here: no per-service Dockerfile, no per-binary Dockerfile, no "server has its own build because it needs X". One file, one parameter.
2. Tests - on PR, on push, and nightly
This is the long one. It runs on every pull request, on push to the main branch, and nightly on a schedule.
| # | Step | What it's for |
|---|---|---|
| 1 | build | fail fast before spending anything on infrastructure |
| 2 | linter | style and static checks |
| 3 | bring up test containers | one script, shared containers, before the run |
| 4 | run tests with coverage | the actual pass |
| 5 | time budget | a separate step that fails when the run gets too slow |
| 6 | tear down containers |
if: always() - runs even when the tests failed |
| 7 | coverage report | produce the number |
| 8 | threshold check | compare against the declared minimum |
| 9 | coverage ratchet | the number may not go down |
Two of these deserve more than a table row.
Containers are shared, one of each type per run. They're started by a single script before the pass, not per test, not per package, not per worker. Isolation between tests is done with data and schemas, not by spawning another database. And the teardown step carries if: always(), because the failure mode you're guarding against is exactly the interesting one: tests failed, so the job short-circuits, so the containers leak, so the next run is slower and the runner eventually falls over. A teardown that only runs on success isn't a teardown.
The ratchet. The threshold check compares coverage against a declared minimum. The ratchet compares the declared minimum against its own previous value: you can raise it, you cannot lower it, and an attempt to lower it fails the PR. It's a one-way valve on a number that otherwise decays quietly, one "I'll fix the tests after" at a time.
There's also a separate scheduled job: a run with the race detector plus the volume tests, with -p 1. That last flag is not decoration - packages run one at a time so nothing else is competing for the machine, which is the only condition under which comparing ns/op across runs means anything. Race and volume don't belong on the PR path; they belong on a schedule where they can take as long as they take.
3. Deploy - manual, two parameters
Deployment is a manual pipeline run with exactly two inputs:
| Parameter | Values |
|---|---|
| version | the tag to ship |
| which services |
server, worker, all
|
After that it's one step, the same step for every service: push the image of that version into the chosen environment.
The point of "manual" here isn't ceremony. It's that the moment of shipping is a decision, and I want a person to make it - but I want that person's entire contribution to be filling in two fields, not remembering a sequence of commands.
Where the pipelines learn what to ship
Now the part that makes "identical for every service" possible instead of aspirational.
None of these pipelines contains anything service-specific. They can't, because they don't know anything about the service that isn't in the manifest:
service:
name: <service>
version: 0.1.0
daemons:
- name: server
handlers: [grpc]
- name: worker
handlers: [scheduler]
That declaration is the source for all three:
| Pipeline | What it reads off the manifest |
|---|---|
| build | the list of daemons → the build matrix (--build-arg BINARY= per daemon) |
| deploy | the same list → the allowed values of the "which services" parameter |
| tests | nothing service-specific at all - it's the same pass everywhere |
Declare a daemon and the build matrix grows a row and the deploy parameter grows an option. Nobody edits a pipeline. Nobody edits a pipeline per service, which is the thing that turns "one path for everything" from a slogan into a property.
The inverse is the useful test of whether this actually holds: if a service needed a special case in the deploy pipeline, the one path would be over. So special cases either get folded into the general form or they don't get built - and yes, that's a real constraint, not a free win. More on that below.
The composition of binaries, incidentally, is not a decision the pipeline makes and not one an implementer makes either. New background work goes into an existing daemon. A new cmd/* is an owner-level decision, written as a prohibition into the instructions everyone works from - because "I'll just add a binary" is how a fleet grows things that nothing deploys.
What's left to a human
Three decisions. Not "three that I could think of" - three, and this is the complete list.
| Decision | Why it stays human |
|---|---|
| environment | which contour receives this version |
| secrets | what the service is handed at runtime |
| tag | which version exists at all |
The third one has the strictest rule attached, and it's a rule about people, not about tooling:
- One person sets tags. Not a bot, not whoever's finishing the work.
- Pseudo-versions are forbidden. Not "discouraged".
- The service carries exactly the tag it was given. If a dependency doesn't resolve because the tag isn't there yet, the answer is to wait for the tag, not to substitute something that resolves.
That last line is the one that took an incident to learn. The alternative - "resolve it now with a pseudo-version, fix the tag later" - produces a green build carrying a version nobody chose. Which is the same failure class as the dev at the top of this article: a build that's green and lying about what it is.
What isn't in the pipeline
Two honest caveats, because "one path for everything" is easy to oversell.
There is no wait-for-health step in the deploy pipeline. None. The platform serves /health and /ready, and the post-deploy check is done by the environment, not by the pipeline that pushed the image. So the pipeline finishing does not mean the new version is up and serving - it means the image was pushed. That's a real gap between what the green check mark implies and what it guarantees, and I'd rather name it than let a reader assume it's covered.
The coverage threshold is currently 0. The ratchet is wired up and works - it can only go up, and it'll fail a PR that tries to lower it - but the bar itself hasn't been set. Actual coverage is 86.7%. So the mechanism is real and the number is not yet doing any work; the honest description is "the valve is installed, the pressure hasn't been dialled in".
There's a third thing worth stating as an absence: there's no deploy UI inside the service. Deployment is the parameterised pipeline run described above, and that's the whole of it - the improvement over the previous state is that it's a form with two fields instead of a remembered sequence of commands.
┌──────────────────────────────────────────────────────────────────┐
│ no wait-for-health step in the deploy pipeline │
│ the pipeline finishing means the image was pushed, not that │
│ the new version is up and serving │
├──────────────────────────────────────────────────────────────────┤
│ the coverage threshold is currently 0 │
│ actual coverage is 86.7% — the valve is installed, │
│ the pressure has not been dialled in │
├──────────────────────────────────────────────────────────────────┤
│ no deploy UI inside the service │
│ a form with two fields instead of a remembered sequence │
│ of commands │
└──────────────────────────────────────────────────────────────────┘
What it costs
I don't think this is free, and the costs are specific.
Special cases have to fit or not exist. The moment one service needs something the shared pipeline doesn't do, there are two options: generalise the pipeline for everyone, or don't do the thing. There isn't a third. That's a real limit on what a single service is allowed to be, and it's the price of the pipelines having no service-specific knowledge in them.
Manual deploy means a human in the chain. Two fields is a small ask, but it's still a person who has to be awake, available, and correct about which version goes where. Everything that follows from having a human in a delivery path - timezones, bus factor, typing the wrong value in a dropdown - follows here.
The ratchet can't be lowered even when lowering is right. A one-way valve is only good while every direction it blocks is a direction you didn't want to go. There are legitimate reasons for coverage to drop - deleting a heavily-tested dead subsystem, for instance - and the mechanism doesn't know the difference between that and laziness. It'll fail the PR either way.
And the version check is a check, not a guarantee of correctness. It proves the symbol exists and the string lands in the binary. It doesn't prove the string is the tag you meant.
The one conclusion
If there's one line to take from this: the pipelines shouldn't know anything about the service that the service didn't declare. Everything mine know - which binaries exist, what can be deployed - comes off one file. And where something can be wrong without producing output, like a linker flag pointed at nothing, the fix is a check that produces output, not a rule asking people to be careful.
That's my current answer and my current bill for it. If you do this better, if you've been through it already, or if you look at it and see it differently - I'd like to hear how it's solved on your side, and specifically what broke on the way there.
Operations out of the box - Part 4.
Next: why generation and operation aren't two problems - the case that a service you can't ship out of the box was never generated properly in the first place.
Top comments (0)