Two binaries. One layout. Four ways to make a new unit of work exist in either of them - and what each way costs.
👋 I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live
PHP monolith into Go services. Earlier parts of this series were about the manifest as the single
declaration a service makes about itself, the environment-variable catalog that falls out of it,
the metrics that arrive with the runtime, and deploy as a consequence of the same file. This part
is the seam underneath all of them: a new handler or a new consumer exists in the repository -
now how does it end up running? Notes:
github.com/brilliant-almazov.
This is what I do on one codebase, with the reasons and the price. You may already do it better,
you may have been here years ago, or you may look at it and disagree - all three are useful to me.
Two binaries out of one image
The service declares its daemons in the manifest, and there are exactly two of them:
service:
name: <service>
version: 0.1.0
daemons:
- name: server
handlers: [grpc]
- name: worker
handlers: [scheduler]
infra:
postgres:
- name: main
migrations: false
tx:
- name: main
pool: main
grpc:
- name: api
daemons: [server]
scheduler:
- name: main
daemons: [worker]
resources:
- type: messaging
name: main
That file is not documentation. Three things read it:
-
The build. One
Dockerfile,--build-arg BINARY=, one image per daemon, pushed on avX.Y.Ztag with the version and commit linked in through ldflags. -
The deploy. A manual pipeline run with two parameters: which version, and which services -
server,worker, orall. Those three values exist because thedaemons:list has two entries. -
The runtime. Resource declarations turn into environment-variable names by one convention,
<TYPE>_<NAME>_<FIELD>:postgres:mainbecomesPOSTGRES_MAIN_DSN,grpc:apibecomesGRPC_API_PORT,messaging:mainbecomesMESSAGING_MAIN_RABBITMQ_URL.
┌────────────────────────────────────────────────────────┐
│ the manifest │
│ daemons: │
│ - name: server handlers: [grpc] │
│ - name: worker handlers: [scheduler] │
└────────────────────────────────────────────────────────┘
│
├──▶ the build one Dockerfile, --build-arg BINARY=,
│ one image per daemon, pushed on a tag
├──▶ the deploy a manual pipeline run: version, and
│ server | worker | all
└──▶ the runtime POSTGRES_MAIN_DSN · GRPC_API_PORT ·
MESSAGING_MAIN_RABBITMQ_URL
Resource names follow one convention: <TYPE>_<NAME>_<FIELD>.
A binary nobody declared is a binary nobody ships.
The composition of that list is the service owner's decision and nobody else's. A new cmd/* is
not something an implementer adds because a new kind of background work showed up - new background
work goes into an existing daemon. That rule is written into every control prompt in this repo, and
it exists because the alternative is a binary that builds fine and has no deploy target: the
pipeline only knows server, worker and all, and a third image nobody declared is an image
nobody ships.
The cost of that rule, stated plainly: worker becomes a mixed bag. The outbound relay, audit
partition retention, derived calculations and the gauge observer all live in one process. They have
different failure modes and different appetites, and they scale together whether that suits them or
not. Splitting them is a decision with a manifest change, a new image and a new deploy target
behind it - not a refactor someone does on a Friday.
Different units, the same shape
The two binaries drive different things. The server drives incoming calls; the worker drives
messages and schedules. The packages that hold them sit in the same layout, one folder per daemon:
internal/daemon/server/<domain> assembly of a domain's Calls
internal/daemon/worker/<...> relay, retention, derived, observe
internal/consumer/<domain> subscriptions
internal/publisher/<domain> outbound publication
On the server side, a domain's assembly is four files, and none of them contains a handler body. The
RPC flow lives in a generic method with a single execution method, Handle(ctx, req) - there are
twenty of those generic RPCs covering create, get, list_revisions, search, update, walk
and the rest. What the domain package holds is a type with a deps field, a NewXxx(deps)
constructor, and a Call() method returning the lambda the generic flow will invoke.
On the worker side, the unit is a consumer bound to a queue, or a job on a schedule. Different
trigger, same skeleton: something is constructed with its dependencies, and then handed to a runtime
that will drive it.
That symmetry is the reason registration can have one shape across both binaries. It is also why the
question "where is this declared" has one answer instead of two. In a tree of 2733 non-generated
.go files across 252 packages, averaging 39 lines each, the assembly point is the only place
where the whole picture of what runs is visible at all. Nobody reads 252 packages to find out what
the worker does; they read one folder.
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
│ server │ │ worker │
│ one unit per domain │ │ one unit per queue or schedule │
│ │ │ │
│ internal/daemon/server/ │ │ internal/daemon/worker/<...> │
│ <domain> │ │ relay · retention · derived · │
│ four files of assembly, │ │ observe │
│ no handler body │ │ internal/consumer/<domain> │
│ Handle(ctx, req) — twenty │ │ internal/publisher/<domain> │
│ generic RPCs │ │ bound to a queue, or run on │
│ NewXxx(deps) -> Call() │ │ a schedule │
└─────────────────────────────────┘ └─────────────────────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ the same three steps on both sides │
│ construct with deps ──▶ register in the set ──▶ the runtime │
│ drives it │
└─────────────────────────────────────────────────────────────────────┘
2733 non-generated files in 252 packages — the assembly point
is the only place the whole picture is visible.
Four shapes registration can take
Here are the four I've actually weighed, with the form each one takes in Go, what it buys, and what
it bills.
1. An explicit list at the assembly point
Everything the binary runs is written out by hand, in one place, in order.
func serverUnits(d Deps) []Unit {
return []Unit{
entity.NewCalls(d),
order.NewCalls(d),
account.NewCalls(d),
}
}
What it buys. The order is visible as source order. Nothing happens by itself - if a name isn't
in that slice, it isn't running, and you can see that without a debugger. When something starts in
the wrong sequence, the fix is moving a line.
What it costs. The list is edited by hand, and there are two of them - one per daemon. A unit
that belongs in both gets added to both, or gets added to one and quietly forgotten in the other.
This is the shape whose failure mode I'll come back to below, because it's the one that bit me.
When it fits. Small sets, and any set where the order genuinely matters and is not derivable.
A list of five is easier to read than any machinery for generating a list of five.
2. Registration through an interface
Every unit implements a common registration contract, and the assembly point iterates a set.
type Unit interface {
Register(ctx context.Context, rt Runtime) error
}
for _, u := range units {
if err := u.Register(ctx, rt); err != nil {
return err
}
}
What it buys. Adding a unit means writing one implementation, and every unit has the same shape -
which matters a lot when the tree already carries generic cores for domain assembly (Builder[Repo,,
Mgr]Domain[Repo, Mgr], HandlerInput[Repo, Mgr]). A uniform contract means the assembly loop is
written once and never touched again.
What it costs. Somebody still builds units. The interface moves the uniformity problem; it does
not remove the list. And the initialization order becomes implicit - it's now the order of a slice
someone constructed elsewhere, which reads like a detail and behaves like a dependency graph.
When it fits. When the set is large enough that per-unit boilerplate is the dominant cost, and
when the units genuinely are peers - no unit needs another one to be up first.
3. Self-declaration at import time
Each package registers itself in an init(), and the binary blank-imports the packages.
func init() {
registry.Add("entity", newEntityUnit)
}
import _ "internal/consumer/entity"
What it buys. Zero edits at the assembly point. The package that owns the behaviour also owns the
fact that it runs.
What it costs. I'll say this one as a risk rather than a convenience: presence and order both
become properties of the import graph. Whether a unit runs depends on whether some file, possibly
in a third package, still has a blank import that a tidy-up tool has no reason to keep. Order depends
on Go's package initialization order, which is a real rule but not one anybody reads a diff against.
And the failure is a runtime failure - the binary starts, the registry is short one entry, and
nothing anywhere says so. Compare that with shape 1, where the same failure is a missing line in a
slice you can read.
When it fits. Plugin-shaped sets where the members are genuinely independent and the cost of one
missing member is low - codec registries, format handlers. Not the set that decides whether cache
invalidation happens.
4. Declared in the manifest, assembled from the declaration
The unit is declared once, declaratively, and the wiring code is produced from that declaration.
daemons:
- name: worker
handlers: [scheduler]
consumers: [entity_invalidation]
What it buys. Declaration and operation become the same fact. That's not hypothetical here: the
manifest already produces environment-variable names, deploy targets and the resource wiring, and
two machine-written snapshots already exist as checked-in artifacts - a catalog of 59 environment
variables (45 from the platform, 14 from the service config, each service variable carrying the
file and line it's defined in), and a snapshot of 67 metric records. Both fail CI on drift. A
consumer declared in the same file would inherit the same property.
What it costs. You need a generator or reflection, and manifest-versus-code drift becomes a new
class of bug - one that is invisible until you build the checker that looks for it. There's a
specific trap in that checker, learned the expensive way on the metrics snapshot: the tool has to be
built at the platform version pinned in the service's module file, resolved from the modules and
compiled at that ref. Build it from the latest platform instead and it compares your snapshot against
a catalog you don't have, and the failure looks both confusing and correct.
Honest status. The generator that would produce daemon assembly this way is designed and not
written: a twelve-iteration specification, with domain assembly listed as fully generated - all four
files, no hand-written bodies - and an explicit ban on starting it without a direct instruction. I'm
describing a shape I've specified and costed, not one I've run.
Four shapes registration can take
──────────────────────────────────────────────────────────────────
1 an explicit list at the assembly point in use
buys order is source order; nothing runs by itself
bills two lists edited by hand — one per daemon
──────────────────────────────────────────────────────────────────
2 a common Register interface uniform
buys one implementation per unit, one shape for all
bills someone still builds the set; order goes implicit
──────────────────────────────────────────────────────────────────
3 init() at import time a risk
buys zero edits at the assembly point
bills presence and order follow the import graph;
the failure is at runtime
──────────────────────────────────────────────────────────────────
4 declared in the manifest, generated from it not written
buys the declaration and the operation are one fact
bills needs a generator; manifest-vs-code drift is a new
bug class
Initialization order is not a detail
Whatever shape you pick, something decides what comes up before what. The dependency chain in this
service is short and completely mundane:
pool → tx registry → repository / manager → domain assembly → handler | consumer | job
Two wrinkles in it are worth naming, because both are places where "it happens to work" and "it is
specified to work" look identical from the outside.
The migrations resource resolves its DSN from the pool's variable. It has no variable of its own.
So the pool declaration has to be real before migrations mean anything - and if you were tempted to
give migrations their own DSN "just to be explicit", you'd have created two sources of truth for one
connection string, which is the same failure as a duplicated metric name in a different costume.
The scheduler resource holds the cron expressions. The service reads no CRON_* variables of its
own. A job's schedule is therefore a property of a declared resource, not of the job's code - which
is what you want, and which is also why the job cannot start before the resource it's scheduled by.
Where this went wrong for real, found during an audit of my own service against the platform: a
scheduler initialized lazily through sync.Once with captured variables, instead of using the
platform's worker init-state store. It worked. It also meant the initialization moment was whenever
the first tick happened to fire, which is not a moment you can point at, order against, or test. The
platform ships worker and worker/initstate precisely so that "this worker is initialized" is a
state you can read rather than an event you hope already happened.
Two more ordering rules that come from operations rather than from the compiler:
-
One publisher per gauge. The outbound-queue and audit gauges are published by
workeronly.serverdoes not publish them, even though the code that could is in the same module. Two processes publishing one gauge produce a graph that jumps between two values depending on which replica was scraped last - it reads as a real oscillation, and it costs an afternoon every time somebody new sees it. That is a registration decision: same code, registered in one daemon and not the other. -
One drainer for the outbound queue. The relay reads rows strictly in
idorder andFOR UPDATE SKIP LOCKEDis deliberately not used, because parallel draining breaks event order, and event order is the thing a consumer relies on to rebuild a correct replica. So the relay is registered once, in one daemon, and "let's run two for throughput" is a data-model change wearing a performance costume.
pool declared as postgres:main; migrations
│ resolve their DSN from the pool variable
▼
tx registry a decorator on mutations; no tx in any
│ signature
▼
repository | manager the manager calls the repository,
│ never copies SQL
▼
domain assembly NewXxx(deps) and Call(); the handler
│ body is not in the domain package
▼
handler | consumer | job the scheduler resource holds the cron;
the service reads no CRON_* variables
┌───────────────────────────────────┐ ┌───────────────────────────────────┐
│ one publisher per gauge │ │ one drainer for the outbox │
│ │ │ │
│ worker publishes them; │ │ rows strictly in id order; │
│ server does not │ │ no SKIP LOCKED │
└───────────────────────────────────┘ └───────────────────────────────────┘
Same code, registered in one daemon and not in the other —
a wiring decision, not a code decision.
The error you want at build time
Here is the one that made me care about this topic.
A cache-invalidation consumer, fully written, registered in no daemon at all. The package
compiled. Its unit tests were green - the handler logic was tested directly, and it was correct. The
subscription was simply never added to either daemon's set, so in production cache invalidation did
not happen. Not intermittently. At all.
Nothing in the toolchain had an opinion about this, and it's worth being precise about why:
- The compiler is satisfied: the package is imported by its own tests, so it isn't even dead code.
- Unit tests are satisfied: they construct the unit directly and exercise it.
- Coverage is satisfied: the package is covered. Coverage measures whether lines ran under test, not whether they run in production.
- The metrics were satisfied too, in the worst way. There is no metric for a subscription that doesn't exist. Broker delivery metrics are per-queue and per-handler, and a handler that never registered has no queue and no series - so the dashboard shows nothing, which is exactly what a healthy idle consumer also shows.
The mirror image of that bug is just as real: registered but not declared. Background work added
to a daemon that the manifest doesn't describe. It runs on your machine and in tests. It has no
deploy target if it needed its own binary, no environment-variable names for the resource it wanted,
and no row in anything that would tell an operator it exists. Where the first bug is silent in
production, this one is silent in the paperwork - and it surfaces at 3am when someone needs to know
what that process is.
Both bugs have the same anatomy: the declaration and the reality are two facts, kept in two places,
with nothing comparing them.
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
│ written, registered nowhere │ │ registered, never declared │
│ │ │ │
│ the package compiles │ │ it runs locally and in tests │
│ its unit tests are green │ │ no deploy target for it │
│ coverage counts the lines │ │ no environment-variable names │
│ no subscription -> no queue │ │ nothing an operator can read │
│ -> no series │ │ │
│ │ │ │
│ invalidation never ran in │ │ silent in the paperwork, │
│ production │ │ not in the logs │
└─────────────────────────────────┘ └─────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ the check that closes both │
│ every registration type appears in exactly one daemon's set │
└─────────────────────────────────────────────────────────────────────┘
Not "is tested", not "is imported" — is in a set that
a daemon iterates.
Reminder, rule, check
The ladder I use for anything in this class is three rungs, and only the third one holds weight.
A reminder lives in one conversation and dies with it. Telling a person - or an agent - "don't
forget to register the consumer" works exactly once, for exactly that consumer.
A rule in a written instruction works for as long as people read the instruction. That's longer
than a reminder and still not a guarantee; the failure mode is quiet and looks like nothing.
A check is the only rung that holds: a linter, a forbidding test, a structural test, a CI drift
check, a blocking hook. In this repository the checks that already exist and already fail builds are
these:
- the environment-variable catalog is regenerated in
--checkmode on any change to a.gofile, the manifest, the module files or the snapshot itself - drift fails the build; - the metrics snapshot has an identically shaped check of its own;
- a forbidding test asserts that a particular row-reading construct appears in exactly one package - it was written after an audit found 44 copies of four row-reading forms in the tree and consolidated them onto one generic core;
- a coverage ratchet that can only go up. Fair disclosure: the threshold currently sits at
0while actual coverage is 86.7% - the mechanism is wired, the bar hasn't been raised yet.
The registration check that follows from this article's bug is the same shape, and it's cheap: assert
that every type implementing the registration contract appears in exactly one daemon's set. Not "is
tested". Not "is imported". Is in a set that a daemon iterates. With shape 4, the same assertion
becomes a comparison between the manifest and the generated assembly, which is why the drift checker
is the interesting part of that shape and the generator is the boring part.
Notice what all four of those checks have in common: none of them are about code being correct. They
are about a declaration and a reality being the same object. That is the entire subject of this
article, and it turns out to be the entire subject of the manifest too.
What this costs
The bill for the way I've landed, without the sales pitch attached.
The assembly point is a file people edit by hand. Two of them, one per daemon. Adding a unit
means touching a list that is not where the interesting code is, and the diff looks trivial, which is
exactly the kind of diff a reviewer skims. That's the shape of the bug above; keeping the shape means
keeping the exposure and paying for a check instead.
Uniformity has a floor. Making every unit implement the same contract means the odd unit - the
one that genuinely needs to come up before another - either doesn't fit the contract, or fits it by
being placed carefully in a slice, which is order-as-a-side-effect and reads as an accident to
anyone who wasn't there.
A rule that forbids new binaries pushes work into worker. I described that above and I'll count
it here as a cost, not a design win: one process is doing relay, retention, derived calculations and
gauge observation, and the reason they share a process is a governance rule about who gets to add a
cmd/*, not a claim that they belong together.
A check is another artifact to keep honest. The env catalog check and the metrics snapshot check
both fail builds, and some of those failures are real drift while others are somebody regenerating on
the wrong platform version. They pay for themselves. They are not free, and pretending otherwise is
how a team ends up with a check nobody trusts and everybody re-runs.
What I picked
An explicit list at the assembly point, with every unit built to the same constructor-and-Call()
shape so the list stays boring, and a check that compares the list against what exists - moving
toward shape 4 when the generator is actually written, because a declaration that is already
producing deploy targets and variable names should be producing this too.
That is a choice with a cost I've just spent four paragraphs describing, not a verdict on the other
three. Shape 2 is better than what I have if your unit count is large. Shape 3 is right for genuinely
plugin-shaped sets and wrong for anything whose absence is silent. Shape 4 is where I think this
ends up, and I'd rather say that than pretend I'm already there.
The one thing I'd argue for regardless of which shape you pick: whatever declares that a unit runs
should be comparable, by a machine, against what actually runs. Every failure in this article -
the unregistered consumer, the undeclared job, the two-source DSN, the lazily initialized scheduler -
is one fact stored twice with nothing checking the copies against each other.
So - how is this solved where you are? I'm specifically interested in two things: whether you use
init()-time registration for anything load-bearing and how that's gone, and whether you have a
check that a written unit is actually wired into a running process. If you generate your wiring from
a declaration, I'd like to know what your drift checker catches most often - I suspect it isn't what
you built it for.
Operations out of the box.
Next in this series: what a service owes the rest of the fleet once it exists - the card that gets
assembled from the same declaration, and who owns it.
Top comments (0)