67 records in the metrics snapshot. The service wrote 6 of them.
👋 I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live
PHP monolith into Go services. Part 1 of this series was about the manifest as the single
declaration a service makes about itself; Part 2 was about the environment-variable catalog that
falls out of it. This part is the same question pointed at observability: what does a service
have to emit before anyone writes a line of instrumentation, and what is it allowed to add on
top? Notes: github.com/brilliant-almazov.
This is what I do on one codebase, with the reasons and the price. You may already do it better,
you may have been here years ago, or you may look at it and disagree - all three are useful to
me.
Start with the measurement, not the opinion
Instrumentation arguments are usually held in the abstract: should the service own its metrics?
how much is too much? I'd rather start with a count, because the count is the argument.
The repository carries a generated snapshot of every metric the service exposes. Not a
README paragraph - a machine-written artifact, regenerated by a tool and checked in. Each record
carries:
| Field | What it holds |
|---|---|
| name | the metric name as it appears on the scrape endpoint |
| type | counter, gauge, histogram |
| help | the help string |
| labels | the label set |
| buckets | histogram buckets, where the type has them |
| source |
platform or service
|
| declared in | the package or file the declaration lives in |
The current snapshot has 67 records:
-
59 with source
platform, -
6 with source
service, - plus one
<dynamic>record, which stands for the dynamic-metric factory rather than for a single fixed name.
That ratio is the whole article. Nine out of ten metric records in this service exist because the
service was started, not because anyone decided to measure something.
67 records in the metrics snapshot
├── 59 platform arrived with the runtime
├── 6 service written by the service
└── 1 <dynamic> the dynamic-metric factory,
not a single fixed name
Nine out of ten records exist because the service was started,
not because anyone decided to measure something.
The reason to keep the snapshot as a file rather than as knowledge is the same reason the
environment-variable catalog is a file: a fact that lives only in a running process can't be
reviewed in a pull request. You can't diff a scrape endpoint. You can diff a snapshot.
What arrives for free
Here is what this service exposes without a single line of instrumentation code in its own
repository. Every item below is emitted by the shared platform library the service imports, and
wired up because the manifest declared the corresponding resource.
Liveness and readiness.
/health
/ready
Both served by the platform on the system port. The service does not implement either handler,
and - worth saying because it surprises people - there is no "wait for health" step in the
deploy pipeline: the endpoints exist so the environment can probe them, and probing is the
environment's job.
Build version.
app_version{version}
One gauge carrying the version string that was linked into the binary. This one is quietly
load-bearing. The linker silently ignores -X pointed at a symbol that doesn't exist: the build
stays green, the version stays dev, and dev is what ends up in the audit trail and in the
trace attributes. Having the version as a metric means the lie is visible on a dashboard instead
of being discovered six weeks later.
gRPC metrics. Request counts, durations and statuses per method, from the platform's
interceptors. The service registers handlers; the platform counts them.
Broker driver metrics.
amqp_delivery_total{queue,handler,status}
amqp_delivery_duration_seconds
amqp_delivery_in_flight
Per-queue, per-handler delivery counts with an outcome label, a duration histogram and an
in-flight gauge. Everything you need to answer "is this consumer keeping up and is it failing"
without the consumer knowing it's being measured.
Outbound relay metrics. The runner that drains the outbound queue publishes: how many rows
are pending, how old the oldest one is, how many were published, how many publish attempts
failed, and how long a cycle took.
Postgres pool metrics. Connection pool state from the driver, per declared pool.
Go runtime metrics. Goroutines, heap, GC - the standard runtime collector.
free with the runtime — 0 lines of instrumentation in the service
────────────────────────────────────────────────────────────────
liveness /health
readiness /ready
build version app_version{version}
gRPC request count, duration, status per method
broker driver amqp_delivery_total{queue,handler,status}
amqp_delivery_duration_seconds
amqp_delivery_in_flight
outbound relay pending · oldest age · published · failures ·
cycle duration
Postgres pool connection pool state per declared pool
Go runtime goroutines, heap, GC
None of that is remarkable individually. What's worth noticing is the shape of how it got
there: the manifest declares a Postgres pool, a gRPC handler and a messaging resource; the
platform turns each declaration into configuration, wiring, and instrumentation in the same
step. There is no separate "add metrics" task, because there is no separate place where metrics
would be added.
What the service adds on top
Six sources in the snapshot are the service's own. The service's domain metric names are declared
in one place, and there are thirteen of them - the full set, with the neutral svc_ prefix
standing in for the real domain prefix:
| Name | Type | Labels | Meaning |
|---|---|---|---|
svc_outbox_pending_rows |
gauge | — | rows waiting to be published |
svc_outbox_oldest_age_seconds |
gauge | — | age of the oldest row in the queue |
svc_outbox_relay_cycle_duration_seconds |
histogram | — | duration of one SELECT → publish → DELETE cycle |
svc_outbox_published_total |
counter | type |
messages published |
svc_outbox_redelivered_total |
counter | type |
publish succeeded, DELETE didn't - the row went out a second time |
svc_outbox_publish_failures_total |
counter |
type, reason
|
publish failures; reason is one of connection, channel, nack, timeout
|
svc_resolve_duration_seconds |
histogram | inherited |
duration of value resolution |
svc_resolve_subjects_per_page |
histogram | — | how many subjects a page returned - catches "a page of 5000 items" |
svc_validation_rejected_total |
counter |
rpc, reason
|
validation rejections by reason |
svc_audit_records_written_total |
counter | subject_type |
audit records written - reconciled against the number of mutations |
svc_audit_partition_oldest_age_days |
gauge | — | age of the oldest audit partition not yet dropped |
svc_audit_partition_dropped_total |
counter | — | partitions dropped by the retention job |
svc_derived_run_duration_seconds |
histogram | calculation |
duration of a background calculation |
Two of these are worth pulling out, because they're the ones that justify the whole set.
svc_outbox_redelivered_total
The outbound queue is a table: one row per domain event, deleted after a successful publish. The
relay reads rows in id order, publishes, deletes. Delivery is at-least-once - the broker does
not deduplicate, and idempotency by event id is the consumer's obligation, written into the
contract comment of the event.
Which means there's a window: publish succeeded, the process died before DELETE landed, the row
gets published again with the same message id. That is normal behaviour, not an incident. The
incident is a spike of it.
You cannot see that distinction in any platform metric. The broker driver knows a message was
delivered; it has no idea it was the same row twice. So the service counts it itself, and the
alert is written against the rate, not against the existence.
SELECT read rows in id order
│
▼
publish the broker has the message
│
▼
DELETE the process died before it landed
│
▼
published again same message id
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
│ at-least-once │ │ a spike of it │
│ │ │ │
│ a redelivery is normal │ │ is the incident — the alert │
│ behaviour │ │ is on the rate, not on the │
│ │ │ existence │
└─────────────────────────────────┘ └─────────────────────────────────┘
The driver knows a message was delivered.
It cannot know it was the same row twice.
svc_resolve_subjects_per_page
A histogram of how many subjects a single page returned. This one exists because of a specific
failure mode: the request is valid, the handler is fast per item, the latency histogram looks
fine - and somewhere a caller is pulling pages of five thousand elements. Per-request duration
hides it. A distribution of page sizes doesn't.
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
│ duration histogram │ │ svc_resolve_subjects_per_page │
│ │ │ │
│ the request is valid │ │ a distribution of page sizes, │
│ the handler is fast per item │ │ not of durations │
│ the latency histogram looks │ │ │
│ fine │ │ 10 · 50 · 100 · 500 · 1000 · │
│ │ │ [5000] │
│ a caller pulling 5000 at a │ │ │
│ time is invisible │ │ one caller lives in the last │
│ │ │ bucket │
└─────────────────────────────────┘ └─────────────────────────────────┘
Each domain metric measures something the platform
structurally cannot know.
That's the general pattern for the six sources the service added: each one measures something
the platform structurally cannot know - a domain-level retry, a domain-level reason, the
duration of a loop the platform doesn't own.
Six rules for adding a metric
Adding a metric to this service is deliberately harder than adding the code that would emit it.
The rules, each with the reason it exists:
Six rules for adding one
────────────────────────────────────────────────────────────────
1 low cardinality the tenant identifier never becomes
a label
2 one owner per gauge worker publishes · server does not
3 names declared once a name is a contract with dashboards
and alerts
4 reason is classified errors.Is(err, ErrConnection),
never parsed out of a message
5 a layer, not a answerable by the other 59 — then it
replacement is not added
6 pull, never push no client, no address, no credentials
1. Low cardinality - the tenant identifier never becomes a label
There are many tenants. A tenant label multiplies every series it touches by the tenant count,
and the multiplication doesn't stop when someone onboards a hundred more.
The per-tenant cut is a real need and it has a real home: queries against the data. Metrics
answer "is the system healthy"; the database answers "what happened for this tenant". Conflating
the two is how a metrics bill becomes a line item.
2. One owner per gauge
The queue gauges and the audit gauges are published by the worker binary only. The server
binary does not publish them, even though the code that could is right there in the same module.
Two processes publishing the same gauge produce a graph that jumps between two values depending
on which replica was scraped last. It looks like a real oscillation. It is a reporting artifact,
and it costs an afternoon every time someone new sees it.
3. Metric names are declared in exactly one place
Names and label keys live in two types - Names and Labels. There is no second copy of a name
literal anywhere in the repository.
// one declaration; every emitter references it
metrics.Names.OutboxPendingRows
metrics.Labels.Reason
The point isn't tidiness. A metric name is a contract with things outside the repository -
dashboards, alert rules, queries someone saved. A string literal copied into a second file is a
contract that can drift from itself, silently, in a diff that looks like a rename.
4. The failure reason is not parsed out of an error message
svc_outbox_publish_failures_total carries reason with four values: connection, channel,
nack, timeout. Those are produced by a classifier that compares the error against a list of
sentinels with errors.Is - not by matching on the error's text.
switch {
case errors.Is(err, ErrConnection):
return reasonConnection
case errors.Is(err, ErrChannel):
return reasonChannel
// ...
}
Two properties follow. The label set is closed - it cannot grow to whatever a driver
maintainer decided to write in a string this month, so cardinality is bounded by construction.
And it survives wrapping: errors.Is walks the chain, a substring match on a formatted message
does not.
5. Domain metrics are a layer on top of the platform's, never a replacement
Nothing in the domain set duplicates a platform metric. The platform has no notion of
redelivery, no classification of a publish failure's cause, no duration for a loop the service
owns. Those three gaps are the entire justification for the domain set.
The inverse test is the useful one: if a proposed metric can be answered by something already in
the 59, it doesn't get added.
6. The collection model is pull, and the application never pushes
An agent scrapes the system port with a bearer token and remote-writes into the metrics store.
The application has no client for the store, no address for it, no credentials, and no code path
that sends anything anywhere. Traces leave over OTLP; metrics are taken, not given.
The practical consequence is that a metrics backend outage is not an application outage, and
"the exporter is misconfigured" is never a class of bug this service can have.
Snapshot drift fails CI
The snapshot is only worth having if it can't quietly stop being true. So there's a check, built
the same way as the check on the environment-variable catalog:
- the generator runs in
--checkmode in CI; - it runs on changes to any
.gofile, the manifest, the module files, or the snapshot itself; - it is built at the platform version pinned in the service's module file - resolved from the modules, fetched at that ref, compiled from there.
That last point took a bug to learn. If the check tool is built from the latest platform instead
of the pinned one, it compares the service's snapshot against a catalog of metrics the service
doesn't have, and the failure is both confusing and correct-looking.
Locally, regeneration is one script. It also reverts the file when the only change is the
generation timestamp - otherwise every branch carries a one-line diff that means nothing and
every review learns to skim past the snapshot, which defeats the point of checking it in.
Where the dashboard lives (not here)
There are no dashboard files and no alert files in this repository, and there won't be. No
dashboards/ directory, no JSON, no rules YAML.
The dashboard is assembled by a generator alongside the rest of the fleet's dashboards, and it
obeys the fleet's style rules rather than whatever looked good on the day:
- colours come only from the classic palette;
- gradients off;
- fill off;
- per-series colour overrides are not allowed;
- stat panels use three thresholds - green / yellow / red;
- timeseries are clean lines;
- uptime uses one duration format on every panel.
Those rules read as fussy until you've compared two services during an incident and found that
"red" meant different things on each. A style rule in a monitoring fleet is a decoding rule:
it's what lets someone who has never seen this service's dashboard read it correctly in the first
ten seconds.
Nine alert rules over these metrics are described in text and created on the fleet's
monitoring side, not in the service repository.
┌───────────────────────────────────┐ ┌───────────────────────────────────┐
│ in the service repository │ │ with the fleet, never here │
│ │ │ │
│ the metrics snapshot, │ │ the dashboard, from the fleet's │
│ 67 records │ │ generator │
│ metric names in one type, │ │ nine alert rules over these │
│ label keys in another │ │ metrics │
│ a CI check built at the pinned │ │ the style rules: classic │
│ platform version │ │ palette, no gradients, no fills │
│ │ │ │
│ │ │ no dashboards/ · no JSON · │
│ │ │ no rules YAML │
└───────────────────────────────────┘ └───────────────────────────────────┘
A renamed metric then breaks in one place, loudly.
The reasoning is the same one that keeps metric names in a single type. A dashboard checked into
a service repository has two failure modes and both are common: it drifts from the metrics it
draws, and it drifts from every other dashboard in the fleet. Keeping the definition next to the
generator means a metric that gets renamed breaks in one place, loudly.
What this costs
I'd rather write the bill than the sales pitch.
The domain set has to be defended. Six service-sourced records is not a natural resting
state - it's a number that only stays small because every proposal to add a label gets pushed
back on. "Just add the tenant id, it'd be so useful on the graph" is a reasonable sentence said
by a reasonable person, and saying no to it is a recurring, slightly unpopular tax.
You can't add a metric locally. One declaration site for names means a developer who wants a
quick counter inside one package cannot have one - they have to touch the shared Names type,
which means a review, which means a conversation about whether the metric should exist. That
friction is the feature, and it is genuinely friction.
Some cuts you want on a graph move into queries. Per-tenant, per-client, per-anything-with-
many-values: not on the dashboard. When someone asks "which tenant caused the spike", the answer
is a query against the data, written at that moment, not a panel someone can look at. That is
slower during an incident. It is the trade I picked, and it's fair to call it a cost rather than
a design win.
The snapshot is another artifact to keep honest. It fails builds. Some of those failures are
real drift and some are someone regenerating on the wrong platform version. The check pays for
itself, but it isn't free.
The one conclusion
The interesting number isn't 67, and it isn't 6. It's that the split between them was never
decided per service. A service declares its resources in a manifest; instrumentation for those
resources arrives with them; and the only thing left to decide is the short list of things the
platform structurally cannot know - which turns out to be redelivery, failure cause, page size
and a few durations.
Everything else is a consequence, and consequences don't need meetings.
So - how is this solved where you are? I'm particularly interested in two things: where you drew
the line between platform-provided and service-provided metrics, and what broke when you got it
wrong. If you keep dashboards in the service repository and it works, I'd like to hear why it
works for you, because I've only seen it fail. And if you've already been through the tenant-label
cardinality argument, I'd like to know which side you landed on.
Operations out of the box - Part 3.
Next: deploy as a consequence of the same manifest - three pipelines that are identical for every
service, and the two parameters a human still gets to choose.
Top comments (0)