Short answer: for a small logistics business, choose the app logging platform whose setup preserves a stable delivery-attempt record before, during, and after a rollback; hosted logs usually reduce operational ownership, while a self-hosted stack buys control at the cost of making storage, upgrades, and recovery part of the team's job.
The hard constraint is not search speed. A notification release can change retry timing, provider selection, or error classification, and a rollback changes the code without undoing messages already attempted. If the old and new releases emit incompatible records, operators cannot tell whether a failed delivery should be retried, suppressed as a duplicate, or reconciled by hand. The easiest setup is therefore the one a junior developer can deploy without weakening the audit trail.
That sounds stricter than ordinary application logging because it is. Delivery is an externally visible side effect.
What does rollback safety require from app logging?
Start with an append-only event shape at the application boundary. Each delivery attempt needs an event identifier, notification identifier, attempt number, release identifier, outcome, and timestamp. A correlation identifier is useful for navigation, but it must not substitute for a stable notification identity: one traces execution, while the other supports idempotency and reconciliation.
The release identifier matters during a rollback. Suppose release notify-42 emits provider_timeout, then release notify-41 resumes traffic and calls the same condition temporary_failure. Search can find both strings, but an audit procedure cannot safely count them as equivalent unless the schema defines that relationship. Keep the wire vocabulary stable across adjacent releases, or emit an explicit schema version and maintain the reader for both versions through the rollback window. Compliance retention limits may also constrain how long raw records and recipient attributes can remain available; those limits belong in the design review, not in an after-the-fact dashboard setting.
Treat the logging appender as a boundary with a contract rather than incidental output. The Logback appender documentation is Java-specific, while the architectural lesson applies more broadly: an appender receives events and sends them to a destination. For this Go service, that boundary should be small enough to test without any logging platform present.
One rule prevents several ugly incidents: the log event reports a completed decision; it never becomes the mechanism that performs the delivery. If log ingestion is unavailable, the notification transaction still follows its defined policy. If delivery fails, its durable business state still drives retry eligibility. Logs explain the state transition; they don't own it.
How should a junior developer compare hosted and self-hosted app logging?
Run the same rollback drill against each candidate instead of comparing feature grids. Hosted logs as a category and Datadog place the ingestion and storage service outside the application team's deployment boundary. A self-hosted ELK deployment puts Elasticsearch, Logstash, and Kibana inside that boundary, so the business owns their capacity, upgrades, access policy, backup, and recovery. Those are deployment-responsibility differences, not a ranking.
Use a small scorecard after the drill:
| Decision test | Evidence to collect | Rollback concern |
|---|---|---|
| Schema continuity | Queries return both adjacent schema versions | Old code must remain readable |
| Duplicate analysis | One notification ID groups every attempt | Retries must not look like new work |
| Ingestion isolation | Delivery policy is unchanged when the sink is absent | Telemetry must not create side effects |
| Access and retention | Documented roles and deletion window | Recipient data has compliance limits |
| Recovery ownership | Named person and tested procedure | Search must survive the release event |
For the smallest team, a hosted option is often easier to set up because fewer logging components sit in its operational boundary. The catch is that externally managed retention, access controls, export behavior, and service limits still need verification against the business's audit obligations. Datadog is one hosted candidate, not a default answer. Self-hosted ELK is not suitable when nobody can own Elasticsearch recovery and the ingestion path during an application rollback; stick with a hosted service in that case. Conversely, keep ELK on the shortlist when the team is already accountable for those components and needs that deployment control. Your mileage may vary because “easy” changes once an organization already has an on-call rotation and a tested recovery process.
Notice what is missing from the scorecard: a promise of exactly-once log delivery. The exactly-once mindset is still useful, but apply it to identifiers and reconciliation rather than assuming a transport can erase every duplicate. A repeated event identifier can be detected. An event with no stable identity cannot be repaired confidently.
Implement the audit event before choosing a destination
The application code should depend on a narrow sink. This example writes newline-delimited JSON to any io.Writer, which makes local development and tests straightforward; a production adapter can send the same event shape to the selected destination without changing delivery logic.
package auditlog
import (
"encoding/json"
"fmt"
"io"
"sync"
"time"
)
type DeliveryEvent struct {
EventID string `json:"event_id"`
NotificationID string `json:"notification_id"`
Attempt int `json:"attempt"`
Release string `json:"release"`
Outcome string `json:"outcome"`
SchemaVersion int `json:"schema_version"`
OccurredAt time.Time `json:"occurred_at"`
}
type Sink interface {
Record(DeliveryEvent) error
}
type JSONLineSink struct {
mu sync.Mutex
w io.Writer
}
func NewJSONLineSink(w io.Writer) *JSONLineSink {
return &JSONLineSink{w: w}
}
func (s *JSONLineSink) Record(event DeliveryEvent) error {
if event.EventID == "" || event.NotificationID == "" || event.Attempt < 1 {
return fmt.Errorf("invalid delivery audit event")
}
s.mu.Lock()
defer s.mu.Unlock()
return json.NewEncoder(s.w).Encode(event)
}
The mutex prevents two goroutines from interleaving bytes in this particular writer adapter. It does not claim distributed ordering, durable storage, or deduplication. Those properties require explicit decisions at the destination or in the surrounding business system. Keep that distinction visible in code review — otherwise a clean JSON line can be mistaken for a durable ledger entry.
Do not log a recipient address, message body, or provider credential merely because it helps debugging. Prefer opaque business identifiers, then document who can resolve those identifiers and under what authorization. I'm not sure which retention period is appropriate without the company's jurisdiction, contracts, and data classification; the answer should come from those constraints and be verified by the responsible compliance owner.
Test the rollback, not just the logger
A unit test should lock down the event schema, but the release gate needs a two-version exercise. Deploy the candidate release to a small traffic slice, record attempts, roll it back, send another attempt for the same notification, and verify that one query reconstructs both decisions in order. Then repeat with the sink disconnected and confirm that the notification service follows its documented failure policy. This is longer than checking that a dashboard received “hello,” because the dangerous bugs live between releases: renamed fields, reset attempt counters, locally generated identifiers that collide after restart, or retry decisions that cannot be joined back to the original notification.
Here is a compact contract test for the adapter. The exact timestamp and IDs are fixed so a failed assertion produces useful evidence rather than a moving snapshot.
package auditlog
import (
"bytes"
"testing"
"time"
)
func TestJSONLineSinkPreservesRollbackFields(t *testing.T) {
var output bytes.Buffer
sink := NewJSONLineSink(&output)
event := DeliveryEvent{
EventID: "evt-000042",
NotificationID: "notice-000017",
Attempt: 2,
Release: "notify-42",
Outcome: "provider_timeout",
SchemaVersion: 1,
OccurredAt: time.Date(2026, 8, 22, 9, 30, 0, 0, time.UTC),
}
if err := sink.Record(event); err != nil {
t.Fatalf("record event: %v", err)
}
want := "{\"event_id\":\"evt-000042\",\"notification_id\":\"notice-000017\",\"attempt\":2,\"release\":\"notify-42\",\"outcome\":\"provider_timeout\",\"schema_version\":1,\"occurred_at\":\"2026-08-22T09:30:00Z\"}\n"
if output.String() != want {
t.Fatalf("unexpected event:\n%s", output.String())
}
}
One test is not enough. Add a compatibility fixture from the previous release, a duplicate event-ID case at the ingestion boundary, and an authorization check for the people who investigate delivery failures. The rollback drill should produce a short evidence packet: release IDs, query used, event count by outcome, duplicate count, missing-field count, and the person who approved the result. That packet is the audit trail for the logging change itself.
Keep it boring.
Roll out with one reversible boundary
Begin by emitting the versioned event to the current destination in shadow mode, with no alert or retry decision depending on the new path. Compare counts by notification ID and outcome across one complete operational cycle, investigate mismatches, and only then move saved searches and alerts. Retain the previous adapter until the rollback window closes; removing it in the same release as the migration turns a telemetry change into a one-way door.
The final choice follows from ownership. Select the hosted path when reducing infrastructure duties makes the rollback drill repeatable for the actual team; select self-hosted ELK when the team can demonstrate recovery, access control, retention enforcement, and upgrades under the same drill. The winning setup is the one that leaves an intelligible, reconcilable record after code moves backward.
Top comments (0)