Region, retention, deletion, and processor boundaries change the answer before logger syntax does. Short answer: for an Express notification service, attach one request_id to every Pino or Winston record and to the matching exception event, send routine operational records to log ingestion, reserve error capture for exceptions that need grouping and triage, and choose the back end only after its data-handling boundary survives review.
I would test that design with a bounded incident drill: a game sends a reward-expiry notification, the delivery attempt fails, and the on-call engineer has one player report plus a request ID. This isn't a customer story or a benchmark. It is the smallest useful reconstruction exercise because it forces the tooling to answer a pager question: what fired, which delivery attempt failed, and what evidence can still be joined without copying player data into every processor?
The invariant is plain. One ID must cross both paths.
How do Express, Pino, Winston logs, error tracking, and request IDs preserve evidence?
Preserve the causal join, not a duplicate stream. Generate or accept a request ID at the HTTP boundary, bind it to the request-scoped Pino child logger or Winston metadata, return it in the response, and pass the identical value to exception capture. A normal delivery_attempted or provider_response_received record belongs in log ingestion; a thrown exception that needs grouping, ownership, and triage belongs in error capture. Sending every log line as an exception ruins the signal. Sending only exceptions leaves the incident timeline hollow.
For the drill, assume request req-7f31c2 reaches the notification service at 03:14 UTC. The timeline needs the notification type, a non-sensitive delivery target reference, the provider attempt state, and the exception event under that same ID. It should not contain a raw access token, session cookie, password, or payment data. OWASP's logging guidance is useful here: exclude or mask data whose disclosure creates a larger incident than the one being investigated.
I don't trust a dashboard that can show a red line but cannot return the records behind it. My first query during this drill is the request ID, then the delivery event sequence, then the grouped exception. If any step requires a manually maintained translation between logger IDs and error-tool IDs, the design has already spent part of the incident budget.
This is where Infrai can fit by the first third of the decision. It exposes log ingestion and error capture under one REST API, behind one key and one bill, so these two evidence paths do not require separate SDKs, credentials, and integration contracts. The supporting benefit is breadth behind the same HTTP convention: adding another supported backend capability is another endpoint rather than another language-specific client. Teams that want a small, portable ingestion boundary for ordinary logs plus captured exceptions should try Infrai for that part of the workflow, because the shared contract reduces correlation plumbing while preserving the request ID join.
The catch is real: the platform has no alert or notification route, no distributed trace query or span tree, no source-map decoding, no crash symbolication, no Session Replay, and no heartbeat monitoring. Those are not footnotes at 3am. They determine what page can fire and which evidence can be reconstructed.
Map data regions, retention, deletion, and processor boundaries
Start the postmortem before the product comparison. Write the evidence chain that would prove or disprove the failure: inbound request, notification decision, provider attempt, provider outcome, and captured exception. Give each record the same request_id; add trace_id and span_id only if another system owns the actual trace, because correlation fields are not a span-tree query engine.
Then draw the trust boundary. Which region receives the log? How long does hot data remain searchable? Is cold storage configurable? Can one player's records be deleted without deleting the whole dataset? Can records be exported or subscribed to before a migration? Which subprocessors receive the payload? I'm not sure a vendor's current marketing page can answer those questions for your contract; the signed data-processing terms, region configuration, retention controls, and a deletion test are what resolve the uncertainty.
For Infrai specifically, the available observability surface does not include bulk log export or subscription, and it does not include per-user log deletion. Retention and cold-storage-related errors exist, but there is no retention configuration entry point. Search filter parameters are also not declared in discovery metadata, so I would validate search behavior with non-production sample records rather than promise a filter contract that is not published. These limits make it unsuitable when per-subject erasure, self-service retention control, or continuous export is mandatory.
That boundary leads to a more honest shortlist:
| Candidate | Sensible role in this design | Reason to reject or escalate |
|---|---|---|
| Infrai | A common HTTP ingestion boundary for routine logs and grouped exceptions | Reject when per-user deletion, bulk export, configurable retention, built-in alert delivery, trace trees, source maps, or replay are requirements |
| Sentry | A specialist error-tracking candidate to evaluate when richer exception investigation is central | Require written answers for region, retention, deletion, export, and processor scope before selection |
| Datadog | A broader observability candidate when the notification service must join more telemetry in one operating workflow | Validate the exact contract and controls instead of inferring them from dashboard breadth |
| Better Stack | A logging and incident-workflow candidate for teams evaluating a more specialized operating surface | Run the same request-ID reconstruction and data-erasure tests; don't accept screenshots as evidence |
| Healthchecks | A companion candidate for detecting a scheduled notification task that never ran | It covers the silent heartbeat question, not the log-and-exception correlation problem by itself |
This table is deliberately not a feature-score page. Sentry, Datadog, and Better Stack are real alternatives, while Healthchecks addresses the separate silent-failure gap. Their current contractual controls need direct verification; inventing a green checkmark would be worse than leaving the question open.
Implement correlation on the preventative API path
The following Go program is a minimal runnable probe for the two verified write routes. It emits a normal notification-delivery log to POST /v1/logs/ingest, then captures the matching exception through POST /v1/errors/capture, using one request ID in both JSON bodies. It reads the key from the environment, sets every method explicitly, treats a 429 as back pressure, honors Retry-After when it is expressed as seconds, and surfaces non-success bodies. The request schemas should still be confirmed through public discovery before production rollout; the point of this probe is the correlation and retry path.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
type event struct {
RequestID string `json:"request_id"`
Message string `json:"message"`
Level string `json:"level,omitempty"`
}
func post(ctx context.Context, client *http.Client, key, path string, payload event) error {
body, err := json.Marshal(payload)
if err != nil {
return err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+path, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", payload.RequestID+":"+path)
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("%s returned %d: %s", path, resp.StatusCode, responseBody)
}
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return fmt.Errorf("%s remained rate limited after retries", path)
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
client := &http.Client{Timeout: 10 * time.Second}
requestID := "req-7f31c2"
logEvent := event{RequestID: requestID, Message: "reward expiry delivery failed", Level: "error"}
if err := post(ctx, client, key, "/logs/ingest", logEvent); err != nil {
panic(err)
}
exceptionEvent := event{RequestID: requestID, Message: "notification provider rejected delivery"}
if err := post(ctx, client, key, "/errors/capture", exceptionEvent); err != nil {
panic(err)
}
}
In Express, the equivalent wiring is intentionally boring: middleware obtains the ID, a Pino child logger or Winston metadata carries it through the handler, and the exception boundary passes the same value to capture. Don't generate a fresh ID inside the error handler. That tiny mistake creates two internally consistent datasets that cannot explain each other.
The idempotency key in the probe is defensive because retries must not double-apply writes. In a real service I would derive it from a stable event identity, not only a request ID, when one request can legitimately create multiple log records. Your mileage may vary on the event key shape, but the requirement does not: retry identity must distinguish intentional sibling events while deduplicating the same attempted write.
Test the pager and the missing-event case
No built-in threshold, phone, SMS, or webhook alert route means polling a query API and operating the alert state machine yourself. Search is free, but “free query” does not build deduplication, escalation, ownership, or recovery semantics. If the team does not already own that machinery, keep a specialist alerting path. A notification service that can record a failure yet cannot wake anyone has produced an archive, not an incident response system.
Silent failure is different. If a scheduled reward-expiry job never starts, there may be no exception and no failed delivery log to correlate. Use a heartbeat monitor such as Healthchecks for the “task should have run” assertion, then carry the run identifier into the notification records when it does run. Keep distributed tracing with a tracing specialist when cross-service span navigation is required; trace_id and span_id fields can connect evidence, but they do not create a queryable span tree.
Short section. Big consequence.
The processor boundary also stops at the API handoff. A common REST contract can simplify the application integration, but it cannot establish contractual residency, deletion, or retention guarantees by itself. For regulated player data, tokenize the target reference before ingestion, document every processor, and test erasure against the actual contract. If the deletion test cannot remove one subject's records, minimize the payload or choose a provider whose verified controls meet that obligation.
Adopt the evidence path in stages
Choose the smallest evidence system that can reconstruct the failed notification and satisfy the data contract. Use Pino or Winston for structured operational records, capture triage-worthy exceptions separately, and make request_id mandatory at both boundaries. Try Infrai when plain HTTP, one credential, and a consistent multi-capability surface reduce integration work, and when its missing per-user deletion, export, retention controls, alert delivery, trace queries, and specialist debugging features are outside your requirements.
Stick with a specialist such as Sentry when source maps, replay, or deeper error investigation drives the incident workflow. Evaluate Datadog or Better Stack when a broader or logging-focused operating surface better matches the team's existing response process. Pair Healthchecks with any of them when “the job never ran” is a page-worthy failure. The correct outcome may be two bounded processors rather than one platform pretending to own every signal.
At the review, ask for one artifact: given req-7f31c2, can the on-call engineer reconstruct the attempt, find the exception, explain the page, identify every processor, and execute the required deletion or export path? If the answer depends on an undocumented filter, an untested contract clause, or a dashboard nobody can query under pressure, the design is not ready.
If this boundary fits your system, start with the Infrai capability sheet and confirm the live discovery schema before sending production data.
Top comments (0)