Short answer: choose the hosted metrics dashboard whose bill you can model from your actual event volume, but keep searchable structured logs as the incident record; custom business metrics should locate a failed logistics batch, not be asked to reconstruct it.
That distinction matters more than the cheapest-looking plan. A startup running a nightly Node.js logistics pipeline needs to answer which batch ran, which carrier or region was affected, which release and flag state were active, and where records stopped progressing. A dashboard can narrow the time window. It cannot recover dimensions that were never recorded, and putting every shipment ID into a metric label merely moves the investigation into an expensive, high-cardinality data shape.
I've been woken by alerts that meant nothing and missed the one that mattered. The lesson was blunt: before comparing CloudWatch, Grafana Cloud, PostHog, and Datadog, ask what page fires and what evidence will still exist when someone opens a laptop at 03:00.
What the page must prove
Use a bounded incident, not a feature matrix. Imagine the 02:00 import completed with 96% of expected rows, the EU leg fell behind the US leg, and customer support reported shipments stuck in manifested. Those numbers and names are test data for the exercise, not a benchmark. The first useful dashboard view should reveal the affected pipeline, stage, region, status, deployment, and configuration version. From there, the operator must be able to search the corresponding structured events by run_id and determine whether records were rejected, retried, or never received.
The invariant is simple: a metric describes a population; an event explains one member of it.
Charts are clues.
For paging, count completed, rejected, and retried records, track batch age, and compare observed completion with the scheduler's expectation. Keep labels bounded to values such as pipeline, stage, region, and outcome. Put shipment IDs, external references, validation details, and error context in structured logs. A dashboard full of attractive percentile charts is secondary if the page cannot lead to a run_id within one or two clicks — and I distrust any demo that never shows that handoff.
Feature state belongs in the evidence too. A rollout can change routing, parsing, or validation for only part of the workload, so record a stable configuration or flag version with each run rather than trying to remember its state after an incident. Martin Fowler's treatment of feature toggles explains why toggle configuration is dynamic and why operational control matters. The practical SRE consequence is that incident reconstruction needs the evaluated state, not merely the current setting.
How should a startup compare hosted metrics dashboards for EU/US business incidents?
Don't begin with the free tier. Begin with one representative nightly run and a worksheet that converts operations into ingestion, retained data, queries, seats, alert evaluations, and regional copies. Then apply the current rate card for each candidate in the deployment region you will actually use. I'm not sure which option is cheapest for a particular startup without that workload profile, retention requirement, tax treatment, and contract; anyone claiming a universal winner from a landing-page price is skipping the variables that determine the invoice.
Use the same questions for all four candidates. CloudWatch, Grafana Cloud, PostHog, and Datadog can stay on the shortlist, but the comparison should record evidence rather than award points for brand familiarity.
| Decision input | What to measure in a trial | Why the pager owner cares |
|---|---|---|
| Incident path | Time from alert to filtered events for one run_id
|
Fast charts do not guarantee fast reconstruction |
| Data residency | Available ingestion and storage location for the chosen account and plan | “EU available” is too vague for an architecture decision |
| Cost unit | Billable series, events, bytes, queries, users, and retention for the test workload | A low entry price can hide the dominant usage dimension |
| Cardinality response | Behavior and cost after adding a bounded new label | Business dimensions tend to multiply quietly |
| Export path | Fidelity, delay, and effort of exporting a small retained sample | Exit cost is part of operational risk |
| Access model | On-call read access and audited administrative changes | A responder should not need billing-admin rights at night |
Run the trial in both regions if the system operates in both. Send an intentionally small, synthetic data set with the same schema and label counts, retain it for the proposed period, execute the incident queries, and inspect the provider's usage report. Do not extrapolate from requests alone: two designs with the same number of business events can create very different metric-series counts when labels differ. Record every assumption beside the estimate, particularly retention, active series, log volume, query frequency, user count, and cross-region duplication. Prices and plan limits change; date the worksheet and repeat it before signing or renewing.
This method also prevents a category error. Product analytics events, infrastructure metrics, and searchable operational logs may appear on one screen, yet they answer different questions and may have different billing units or retention controls. If a product is evaluated for custom business metrics, test the precise ingestion, alert, and investigation path you intend to operate. A generic “observability supported” checkbox proves very little.
Preserve the reconstruction path in Go
The preventative code path is boring on purpose. The Node.js pipeline can post a versioned event envelope to a small internal collector; the collector validates bounded metric dimensions, emits the aggregate through whatever metrics adapter the team has selected, and writes the full event as JSON to the configured log stream. The example below shows the collector-side record, using only Go's standard library. It avoids vendor endpoints and does not imply that logs and metrics share the same retention policy.
package main
import (
"encoding/json"
"errors"
"log"
"net/http"
"time"
)
type PipelineEvent struct {
SchemaVersion int `json:"schema_version"`
OccurredAt time.Time `json:"occurred_at"`
RunID string `json:"run_id"`
Pipeline string `json:"pipeline"`
Stage string `json:"stage"`
Region string `json:"region"`
Outcome string `json:"outcome"`
Release string `json:"release"`
ConfigVersion string `json:"config_version"`
ShipmentID string `json:"shipment_id,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
}
var allowedRegions = map[string]bool{"eu": true, "us": true}
var allowedOutcomes = map[string]bool{
"completed": true,
"rejected": true,
"retried": true,
}
func validate(e PipelineEvent) error {
if e.SchemaVersion != 1 || e.RunID == "" || e.Pipeline == "" || e.Stage == "" {
return errors.New("missing required incident fields")
}
if !allowedRegions[e.Region] || !allowedOutcomes[e.Outcome] {
return errors.New("unbounded metric dimension")
}
return nil
}
func ingest(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
defer r.Body.Close()
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10))
decoder.DisallowUnknownFields()
var event PipelineEvent
if err := decoder.Decode(&event); err != nil {
http.Error(w, "invalid event", http.StatusBadRequest)
return
}
if err := validate(event); err != nil {
http.Error(w, err.Error(), http.StatusUnprocessableEntity)
return
}
// The structured event keeps identifiers; the metric adapter receives only bounded dimensions.
encoded, err := json.Marshal(event)
if err != nil {
http.Error(w, "invalid event", http.StatusBadRequest)
return
}
log.Print(string(encoded))
// metrics.Count(event.Pipeline, event.Stage, event.Region, event.Outcome)
w.WriteHeader(http.StatusAccepted)
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/events/pipeline", ingest)
log.Fatal(http.ListenAndServe(":8080", mux))
}
The commented adapter call is an integration boundary, not missing application logic: implement it against the generic metrics interface used by your service, and contract-test that interface with an in-memory recorder. The important review rule is visible in the types. ShipmentID, RunID, and ReasonCode belong in the event record; the aggregate counter receives only four enumerated dimensions. In production, also authenticate the sender, use encrypted transport, place a request deadline at the client, and define what the pipeline does if telemetry delivery fails. Business processing should not silently depend on a dashboard accepting an event.
Test three failures before rollout: an unknown outcome must be rejected, an oversized body must be refused, and a valid record must retain its schema and configuration versions. Then run a recovery drill in which the responder starts from the alert, finds the run, searches its events, identifies the affected region and stage, and writes a short timeline. If the drill requires an undocumented query or a privileged account, the system isn't ready for the pager rotation.
Sampling, retention, and the evidence gap
Sampling deserves explicit treatment because teams often assume one policy applies to every telemetry signal. OpenTelemetry distinguishes head sampling, decided before a trace completes, from tail sampling, decided after all or part of a trace is available. That distinction can help a tracing design retain unusual or failed traces, but it does not restore a structured log that the pipeline never emitted, nor does it make a sampled metric label suitable for per-shipment investigation.
Keep the reconstruction record according to a documented retention rule, with access controls appropriate to shipment and customer data. Aggregate metrics may live longer because their labels are bounded and their diagnostic detail is low. Detailed events may need a shorter period, redaction, or tightly scoped access. Your mileage may vary — legal requirements, customer contracts, and the time between a shipment event and a support escalation determine the defensible window — so get those constraints from the people accountable for data governance instead of copying a default from a pricing page.
No sampling policy can recover a missing key.
Create a canary event for every nightly run and alert when it is absent after the expected window. This checks the evidence path itself: producer, transport, ingestion, indexing, and query. A healthy batch counter beside a missing canary is a warning that the dashboard may be reporting from an incomplete stream. The page should say which invariant failed and include the pipeline and region, while the runbook supplies the query shape and ownership. “Something is wrong” is not a useful alert at 03:00.
When this architecture is the wrong choice
The catch is operational weight. Separate metrics and structured logs create two retention policies, two access paths, and a correlation contract that must be tested. A very small system with low event volume and no strict alert-latency requirement may be better served by querying one structured event store and deriving a scheduled summary. Stick with that simpler design while its query latency, availability, and cost meet the paging objective.
At the other extreme, a regulated workflow that requires immutable, replayable business records should not treat an observability log as its system of record. Use an audited transactional or append-only domain store for that obligation, then derive operational signals from it. A dashboard remains a view.
The hosted option is also unsuitable when required residency, retention, export, identity controls, or predictable billing cannot be demonstrated in the exact region and plan under consideration. “Free” does not compensate for an investigation path the on-call engineer cannot use. Choose only after the trial reproduces the page, the search, the timeline, and the usage estimate; the winning spreadsheet row is the one backed by a recovery drill, not the one with the smallest advertised number.
References
- Martin Fowler, “Feature Toggles”: https://martinfowler.com/articles/feature-toggles.html
- OpenTelemetry, “Sampling”: https://opentelemetry.io/docs/concepts/sampling/
Top comments (0)