For a small SaaS evaluating a Prometheus alternative, send each completed game-data batch through an idempotent push API, then expose reconciled custom metrics to the dashboard; do not treat either a successful push or a successful pull scrape as proof that the nightly pipeline produced correct data. The deciding constraint is signal quality: the service needs to distinguish late, duplicated, partial, and invalid batches without turning every log line into a metric, regardless of whether Grafana or another dashboard renders the result.
TL;DR: For a nightly pipeline, push one bounded completion record per batch into a durable receiver, derive low-cardinality metrics there, and retain structured logs as the audit trail. A pull endpoint remains useful for live process health. The two transport models solve different failure windows, so combining them is often clearer than declaring either one the universal winner.
This architecture decision record uses four boundaries: batch identity, acceptance, reconciliation, and publication. They matter more than the choice of dashboard.
Should a small SaaS push or pull custom metrics?
The concrete workload is a gaming pipeline that transforms a night's events into searchable structured logs and a small custom-metrics dashboard. Operators need to answer questions such as: Did every scheduled shard finish? Which game mode produced rejected records? Is the apparent drop in processed events real, or did telemetry arrive late?
Four invariants keep those answers defensible. First, every batch has a stable identifier derived from the scheduled run and shard, not from an individual retry. Second, accepting an observation never silently replaces a conflicting observation for the same batch. Third, aggregate counts can be reconciled against the accepted batch records. Fourth, dashboard publication cannot erase the underlying audit evidence. A beginner may reasonably start with the visible choice between a Prometheus pull model and a custom push API, but that framing omits the harder question: which observation survives long enough to prove what happened after an ephemeral job exits? A Node.js producer, a Go producer, and a shell-launched batch all face the same boundary. Transport syntax changes; the reconciliation obligation does not.
These are exactly-once effects, not a claim that a network transports a message exactly once. Retries happen. A receiver can acknowledge a request just before the client loses the response, and a scraper can miss a short-lived process between intervals. The design therefore makes repeated delivery harmless and exposes uncertainty rather than hiding it.
Keep the labels restrained. environment, pipeline, game_mode, and a bounded result can support operations; player_id, raw error text, and batch_id belong in logs or a lookup store, because their value sets grow with the workload. OpenTelemetry distinguishes metric instruments and their intended semantics, but an instrument does not rescue a poor attribute model. Cardinality is an architectural input.
Noise compounds.
A compliance-minded retention policy also separates the two records. Metrics answer aggregate operational questions, while structured logs preserve the traceable batch decision with access controls and retention appropriate to their contents. Do not copy player identifiers into metric attributes merely because a dashboard query feels convenient.
Decision: split durable batch facts from live health
The critical path is short. At batch completion, the worker submits one summary containing the stable batch key, counts, timestamps, and a checksum of the summarized output. A receiver validates it, applies an idempotency rule, persists the accepted fact, and returns an acknowledgment. A separate projector turns accepted facts into bounded metrics. The dashboard reads those metrics, while investigators search structured logs by batch key.
The worker also exposes pullable process metrics for memory pressure, queue depth, and current work. Those observations describe a running target and can disappear when the target exits; they are not the ledger of completed overnight work. Separating the paths prevents a common category error: process liveness and business completion have different clocks.
| Decision point | Push completion record | Pull live target |
|---|---|---|
| Natural clock | A batch reaches a terminal state | A process is running at scrape time |
| Main failure boundary | Ambiguous acknowledgment and retry | Missed samples during target absence |
| Duplicate control | Stable idempotency key plus conflict detection | Collector timestamps repeated samples |
| Best signal here | Completed, rejected, and reconciled batch totals | Queue depth and worker health |
| Noise risk | One series or event per unbounded identifier | Scraping many ephemeral or irrelevant targets |
| Audit role | Durable fact can point to detailed logs | Snapshot alone is insufficient evidence |
This comparison does not depend on a particular metrics product. OpenTelemetry provides a vocabulary for metric signals and instruments; it does not require the batch fact and the live gauge to use the same delivery path. Likewise, the dashboard is a view, not the system of record.
For a small operation, simplicity means fewer ambiguous states, not necessarily fewer components. One receiver, one durable table, and one projector may be easier to reason about than teaching every ephemeral worker to retain state until an external scraper happens to observe it. The reverse can be true for long-running services whose useful signals already exist continuously in memory. The trade-off is operational ownership: a durable receiver introduces schema evolution, storage recovery, access control, and projector lag. It is unsuitable when the team cannot operate that state correctly or when losing a best-effort counter has no consequence. In those cases, the pull model's smaller failure surface can be the better decision.
Critical path in Go
The following handler illustrates the contract, including the distinction between a duplicate and a conflict. The storage interface must enforce uniqueness on BatchID; an in-memory check followed by an insert would reopen the race.
package ingest
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"net/http"
"time"
)
type Completion struct {
BatchID string `json:"batch_id"`
Pipeline string `json:"pipeline"`
GameMode string `json:"game_mode"`
Processed uint64 `json:"processed"`
Rejected uint64 `json:"rejected"`
Finished time.Time `json:"finished_at"`
}
type Store interface {
InsertOrCompare(ctx context.Context, batchID, digest string, c Completion) (inserted bool, conflict bool, err error)
}
type Handler struct{ Store Store }
func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
var c Completion
dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10))
dec.DisallowUnknownFields()
if err := dec.Decode(&c); err != nil || c.BatchID == "" || c.Pipeline == "" {
http.Error(w, "invalid completion", http.StatusBadRequest)
return
}
canonical, err := json.Marshal(c)
if err != nil {
http.Error(w, "cannot encode completion", http.StatusInternalServerError)
return
}
sum := sha256.Sum256(canonical)
digest := hex.EncodeToString(sum[:])
inserted, conflict, err := h.Store.InsertOrCompare(r.Context(), c.BatchID, digest, c)
if err != nil {
if errors.Is(err, context.Canceled) {
return
}
http.Error(w, "temporary ingest failure", http.StatusServiceUnavailable)
return
}
if conflict {
http.Error(w, "batch identity conflict", http.StatusConflict)
return
}
if inserted {
w.WriteHeader(http.StatusCreated)
return
}
w.WriteHeader(http.StatusNoContent)
}
Two limits are concrete here.
The 64 KiB body limit is a contract guard, not a throughput claim. A completion summary should remain small; detailed records stay in the structured-log path. The HTTP outcomes also have deliberately narrow meanings: 201 records a new fact, 204 recognizes an identical retry, 409 identifies a conflicting reuse of the batch key, and 503 leaves the client free to retry because no durable acceptance was promised. DisallowUnknownFields makes schema drift visible, although a production evolution policy may instead accept known versioned envelopes. This strictness has a limitation: independently deployed producers cannot add fields casually. A versioned envelope or an explicit compatibility window is required once producer and receiver releases can diverge.
The subtle part is the digest. Replaying an identical body for the same batch_id is success, because the desired effect already exists. Reusing that identifier with different counts is a conflict that should page or enter a review queue; silently choosing the latest payload would destroy the audit trail. Persist the accepted record and its digest in one transaction. Only then may the projector update dashboard-facing aggregates.
A projector can fail after persistence. That is acceptable if it checkpoints progress and can replay accepted records, because the dashboard may briefly lag while the durable fact remains intact. The lag itself should be measured as the age of the newest unprojected record, using a bounded set of attributes. This yields a signal about the observation system without recursively emitting a metric for every failed event.
How do we test signal quality rather than transport success?
Begin with fault cases, since a happy-path request proves little. Send the same completion twice and require one durable fact. Send the same key with altered counts and require a conflict. Cancel the client after the server commits, retry, and require an idempotent acknowledgment. Stop the projector, accept several batches, restart it, and require aggregates to converge without double counting. Finally, terminate a worker before the next scrape and verify that the batch remains visibly incomplete rather than falsely successful.
Reconciliation is the stronger test. For each nightly run, compare the expected shard manifest with terminal batch facts, then compare the sum of accepted processed and rejected counts with the pipeline's output manifest. A dashboard status is complete only after those checks agree. HTTP success means accepted, not reconciled.
Deployment deserves the same caution. Add the completion schema before requiring new fields, deploy the receiver before producers emit the new version, and keep projector rebuilds possible from durable records. An alert should fire on missing expected batches, identity conflicts, projector lag, and reconciliation mismatch. It should not fire once per malformed log line.
Review the noise weekly.
Inspect alert volume, the fraction of signals that led to an action, attribute cardinality growth, and time to reconcile the nightly run. These are decision criteria rather than invented universal thresholds. The acceptable limits depend on the schedule, on-call coverage, and regulatory obligations governing the underlying data. Document those limits beside the ADR and test them as configuration. An easy first assumption is that more labels make the Grafana dashboard more useful; later reconciliation exposes the flaw, because an identifier useful for one forensic search can create a separate metric series for every batch or player. Keep that identifier in the structured logs and link investigations through the stable batch key.
Rejected option and where it still fits
The rejected design exposes only an in-memory Prometheus pull endpoint from each nightly worker. It is attractive because there is no explicit ingestion receiver, and it can be adequate for a long-running service whose metrics describe current process state. For short-lived batch workers, however, absence is ambiguous: the job may not have started, may have completed between scrapes, or may have crashed before exposing a final value. A Grafana dashboard cannot infer which event occurred from a missing target. This is a limitation of the observation boundary in this workload, not a general defect in Prometheus.
The other extreme, pushing every structured log field directly into metric labels, is also rejected. It improves apparent search flexibility at the expense of uncontrolled series growth and muddles two data models. Search belongs over structured records; aggregation belongs over deliberately bounded dimensions. A custom log sink can encapsulate delivery, as the Logback appender documentation demonstrates for that ecosystem, but delivery plumbing does not define the correctness contract.
There is a valid use case for pull alone: stable services with discoverable endpoints, continuously meaningful gauges, and no requirement to prove that a transient batch reached a terminal state. There is also a valid use case for a simple push gateway without a durable fact store: disposable best-effort counters whose loss or duplication cannot affect an operational decision. Neither description matches a nightly gaming pipeline used for reconciliation.
The final choice is architectural, not promotional. Preserve one immutable batch fact, make retries idempotent, project only bounded metrics, and use pull observations for live health. That arrangement gives a small team a dashboard it can trust without asking the dashboard to become an audit database.
Top comments (0)