Short answer: instrument the checkout workflow around one reconstruction contract, then send its metrics, structured events, and correlated errors to replaceable backends while an independent probe tests the customer-visible path. For a small application in 2026, the deciding constraint is not the number of dashboards; it is whether support can reconstruct one failed checkout without guessing, leaking payment data, or treating an uptime check as proof that the transaction completed.
This architecture decision record defines that contract before choosing storage or visualization software. It assumes a customer-support team must answer a precise question: what happened to checkout chk_7f31 between request acceptance and the final outcome? That question makes correlation, bounded cardinality, idempotency, and an auditable event sequence more important than a broad product checklist.
How should a small app stack combine health monitoring, logs, metrics, and errors?
The invariant is simple: every accepted checkout has a stable, opaque checkout_id; every attempt has an attempt_id; every state transition records an event name, event time, outcome, and trace correlation; and repeated delivery does not create a second logical transition. A support agent should be able to distinguish a validation rejection, an upstream timeout, a retry already in progress, and a completed checkout whose response was lost.
That sounds close to ordinary logging, but the failure boundaries differ. A process may die after committing business state and before emitting telemetry. A telemetry exporter may retry and deliver the same record twice. A browser may abandon the page while the server continues. The health endpoint may return success while the checkout dependency is unavailable. None of those boundaries grants exactly-once transport, so the observability contract must tolerate missing, late, and duplicated records rather than implying certainty it does not possess.
Use business identifiers as event attributes where reconstruction needs them, but keep them out of low-cardinality metric labels. Metrics answer whether failures are increasing; event records answer which checkout failed. OpenTelemetry describes metrics as runtime measurements and distinguishes additive counters from measurements such as gauges and histograms. That division supports a compact metric surface: attempt count by coarse outcome, duration distribution by workflow stage, and a current readiness value. A raw checkout identifier would make each metric series unique and defeat that purpose.
The privacy boundary is equally firm. Do not record card numbers, security codes, authorization secrets, session cookies, free-form support text, or complete request bodies. Permit-list fields, hash or tokenize customer identifiers only when the investigation model requires stable correlation, and attach retention and access rules to the event dataset. Compliance scope is an architectural limit, not a log-redaction task deferred until an incident.
No payload dumps.
Decision: separate detection from reconstruction
Adopt four cooperating signals with distinct jobs. An external uptime probe detects whether a customer-like request can reach a shallow public endpoint. Readiness checks decide whether an instance should receive traffic. Metrics detect aggregate changes in rate, error ratio, and duration. Structured event records, linked by trace identifiers, reconstruct the individual checkout.
Keep them separate.
A probe should not create a real order or call a payment processor on every interval. It can test DNS, TLS, routing, and a deliberately bounded application path. Readiness may check only dependencies required to serve new work, under a strict time budget, because a deep dependency tour can amplify an outage. Neither signal replaces a synthetic checkout in a controlled environment, and the synthetic path must use unmistakable test data plus an idempotency key.
| Option | Detection quality | Reconstruction quality | Failure boundary | Decision |
|---|---|---|---|---|
Process logs plus one /health probe |
Finds crashes and broad reachability loss | Weak when records lack stable correlation or transitions | Probe success can coexist with checkout failure | Reject for this job |
| Metrics, structured events, and independent uptime checks | Separates aggregate health from per-checkout evidence | Strong when the event schema and identifiers are stable | Export and storage remain lossy or duplicative | Adopt |
| Full request and response capture | Rich payload evidence | High apparent detail, but excessive sensitive-data and access exposure | Collection becomes part of the compliance boundary | Reject by default |
The adopted design does not require one database. It requires one semantic contract. A small team can retain events in an existing structured log store, expose metrics through an OpenTelemetry-compatible path, and use any scheduler or monitoring service for the external check. Replacement is then an adapter change, while the checkout state vocabulary remains stable.
The limitation of this design is operational breadth: four signal paths create four places to test permissions, delivery, retention, and clock behavior, while a single process log is easier to operate. The trade-off is justified only when individual checkout reconstruction is a real support obligation. It is a poor fit for an ephemeral internal tool with no durable transaction state, and it cannot repair an ambiguous domain model; if the application cannot say which state transition committed, adding telemetry merely records the ambiguity with greater precision. It also leaves gaps by design. A sampled trace may be absent, an aggregate counter may be duplicated near a retry boundary, an external probe cannot prove that every dependency works, and a retention cutoff may remove old evidence. The durable business record therefore remains authoritative, with observability serving detection and reconstruction rather than replacing reconciliation.
That boundary matters.
This also clarifies cost without turning price into the decision. Event volume grows with attempts and transitions; metric volume grows with label combinations; traces grow with sampled spans. Budget those three dimensions separately, set retention by investigative need, and inspect any provider's ingestion-versus-indexing model before assuming that collected data is equally searchable. For example, Datadog's public pricing page distinguishes log ingestion from indexed log retention. That is evidence about a billing boundary, not a reason to select or reject the product.
Critical path in Go
The application should emit a transition only after it knows the corresponding business result, using a deterministic event key so exporter retries can be deduplicated downstream. The following Go example keeps the interface generic and deliberately excludes payment payloads. Its ledger write and event publication are represented as one repository operation: an implementation should use a transactional outbox or an equivalent atomic persistence boundary, rather than claiming that a network call after a commit is exactly once.
package checkout
import (
"context"
"fmt"
"time"
)
type Transition struct {
EventID string
CheckoutID string
AttemptID string
Name string
Outcome string
TraceID string
OccurredAt time.Time
}
type Repository interface {
// CommitOutcome stores business state and its outbox event atomically.
CommitOutcome(context.Context, Transition) error
}
type Metrics interface {
ObserveAttempt(ctx context.Context, outcome string, elapsed time.Duration)
}
type Service struct {
repo Repository
metrics Metrics
now func() time.Time
}
func (s Service) RecordFailure(
ctx context.Context, checkoutID, attemptID, traceID, reason string, started time.Time,
) error {
// reason must come from a permit-listed, low-cardinality vocabulary.
event := Transition{
EventID: fmt.Sprintf("checkout:%s:attempt:%s:failed", checkoutID, attemptID),
CheckoutID: checkoutID,
AttemptID: attemptID,
Name: "checkout.failed",
Outcome: reason,
TraceID: traceID,
OccurredAt: s.now().UTC(),
}
if err := s.repo.CommitOutcome(ctx, event); err != nil {
return fmt.Errorf("commit checkout failure: %w", err)
}
s.metrics.ObserveAttempt(ctx, reason, s.now().Sub(started))
return nil
}
The deterministic EventID is an idempotency aid, not proof of globally exact-once execution. The repository enforces uniqueness, the outbox publisher may deliver more than once, and the consumer applies the same key when inserting into the reconstruction store. Metrics may still be incremented twice around a retry boundary, so reconciliation relies on durable transitions, not on treating a counter as a ledger.
Notice the ordering. If the atomic commit fails, no success-shaped event is emitted. If metric observation fails after the commit, the durable outbox still preserves the reconstructable business transition. This is the same asymmetry a financial ledger needs: the audit trail follows committed state, while aggregate telemetry is useful but not authoritative.
Operational method and acceptance tests
Test the contract as a failure matrix, not as a screenshot of a green dashboard. Kill the process before the database commit, after the commit but before outbox publication, and after publication but before acknowledgement. Inject a checkout dependency timeout. Deliver the same outbox record twice. For each case, assert the business state, the number of logical transition records after deduplication, and the support timeline shown for the checkout identifier.
Three concrete acceptance thresholds keep the design review honest: one logical terminal outcome per attempt_id; zero sensitive payment fields in emitted records; and a bounded vocabulary for metric labels. These are schema properties, not performance benchmarks. The team should set alert thresholds from its own traffic and service objectives rather than copying generic percentages.
Deploy schema changes additively. Consumers should ignore unknown fields, producers should preserve the meaning of existing event names, and a breaking semantic change should receive a new version rather than quietly reusing a field. Keep a sampled fixture for each supported event version so support tooling can be tested before rollout. During an incident, compare the durable checkout state, outbox state, event store, and metric trend in that order; a discrepancy becomes a reconciliation item with an owner and timestamp.
Audit access to the reconstruction view itself. The useful questions are who queried a checkout, which fields were revealed, and whether an export occurred. A technically complete timeline without access accountability merely moves the risk from production logs into the support console.
Rejected option and where it still fits
We reject a single all-purpose stream containing complete request bodies, stack traces, health events, and metric-like counters. It couples retention, access control, cardinality, and query cost; it also tempts operators to infer aggregate health from whatever text happened to be logged. More data can produce less reliable evidence when fields lack a stable meaning.
The simpler process-log approach remains valid for a low-risk internal utility whose failures have no durable business outcome, whose users can reproduce requests safely, and whose operators do not need per-transaction reconciliation. Add a shallow uptime check and a small set of metrics there. Once checkout support must explain a committed outcome across retries, callbacks, and lost responses, the reconstruction contract earns its complexity.
The final selection rule is therefore vendor-neutral: choose components that preserve stable identifiers, accept OpenTelemetry-shaped signals, expose explicit retention and access boundaries, support deduplication of durable events, and let an independent probe observe the public path. Evaluate them with replay and failure-injection tests. The best stack is the one from which a support engineer can produce a bounded, auditable account of one checkout without mistaking telemetry for business truth.
Top comments (0)