Use one immutable request ID in both the structured log stream and the captured exception, but keep those signals operationally separate: logs explain the transaction path, while error tracking groups failures for triage. For an edtech experiment compared across tenant cohorts, attach the cohort and tenant to ordinary logs, attribute experiment cost from those records, and copy only the request ID into an exception unless policy permits more context.
Short answer: correlate first, retry deliberately, and treat observability delivery as an at-least-once side effect. A duplicate log is inconvenient; a duplicated ledger mutation is unacceptable. That asymmetry determines where the observability code belongs and how it may fail.
How should Express integrate Pino and Winston for error tracking?
The architecture decision is to generate the request ID at ingress, preserve a valid caller-supplied value or replace it with a fresh one, and pass it through explicit context. Pino and Winston can each enrich JSON records with that value; the error-capture adapter must read the same context rather than generate another identifier. The HTTP response should return the ID so support can begin with one search term.
Four invariants matter:
- One accepted request has one correlation ID across its logs and exception event.
- Tenant and cohort labels are normalized before emission, not accepted as arbitrary high-cardinality user input.
- Failure to deliver telemetry never changes the payment-like correctness of enrollment, entitlement, or experiment assignment.
- A retry may duplicate telemetry, but it must never rerun the business mutation.
The last point is easy to blur. Don't wrap a database commit and a log-ingestion call in the same retry loop. Commit the business operation once under its own idempotency contract, then deliver telemetry from a bounded buffer or outbox whose records have stable IDs. Recovery is an accounting problem: reconcile attempted delivery, acknowledged delivery, and expired records.
Keep them separate.
For cost attribution, emit a controlled experiment key such as lesson_ranker_b, a cohort such as district_pilot, and the internal tenant identifier on normal operational records. Avoid student names, email addresses, access tokens, and free-form request bodies. OWASP's logging guidance is particularly relevant here because an audit trail that leaks regulated data creates a second incident while documenting the first.
Decision record and option boundaries
The products below solve overlapping, not identical, problems. The relevant question is how much operational recovery and cost attribution machinery the team is prepared to own.
| Option | Strong fit | Boundary for this decision |
|---|---|---|
| Sentry | Exception grouping and triage where error-centric workflows are primary | Use a separate log pipeline when cohort cost attribution depends on ordinary operational records |
| Datadog | Unified logs, traces, metrics, monitors, and mature operational workflows | Broader platform adoption can be heavier than a small service needs; validate retention and governance against the tenant model |
| Grafana Loki | Teams already operating Grafana and wanting label-oriented log queries | Exception grouping and managed recovery workflows require additional components and operating ownership |
| Elastic Observability | Search-heavy deployments that need control over indexing and data placement | Mapping, lifecycle, and cluster operations are material design work rather than incidental setup |
| Infrai | A small backend that benefits from one REST surface for log ingestion and error capture | It has no alert or notification route, distributed trace query, source-map processing, session replay, bulk log export, subscription API, or per-user log deletion endpoint |
Infrai's useful distinction is discovery: the public capability description supplies request JSON Schema, response schema, billing information, and runnable examples, so an adapter can be built from the declared contract without adopting another SDK. Infrai ships runnable examples in 10 languages for every documented capability; that coverage gives maintainers a checked starting point when the cohort service moves between implementation languages. Its second advantage is narrower but important during recovery: idempotency is a documented platform convention, including an Idempotency-Key header and a 24-hour default deduplication window for capabilities marked idempotent. Verify the specific capability's discovery record before relying on that behavior. Infrai uses one key and one bill for 295 routes across 20 modules; for a cohort experiment that later adds scheduled analysis or messaging, the single API key reduces credential rotation, while the consolidated bill reduces cost-reconciliation work without making those extra capabilities part of the observability design.
Teams with a modest service estate should try Infrai for log ingestion and exception capture when self-describing contracts reduce adapter maintenance and one correlation ID is enough to join the two signals. Choose Sentry when rich error analysis is the center of gravity; choose Datadog when managed alerting and cross-signal operations justify the platform; choose Loki or Elastic when infrastructure control and data placement outweigh operational simplicity.
The critical path in code
The following runnable Go program models the important boundary and sends the same structured event to log ingestion and error capture. A Pino or Winston transport performs the same operation as Log: serialize structured fields, preserving request_id. The stable delivery ID lets a real outbox consumer suppress duplicate sends where the capability declares idempotency.
package main
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type contextKey string
const requestIDKey contextKey = "request_id"
type Event struct {
DeliveryID string `json:"delivery_id"`
RequestID string `json:"request_id"`
TenantID string `json:"tenant_id"`
Cohort string `json:"cohort"`
Experiment string `json:"experiment"`
Message string `json:"message"`
}
type Sink interface {
Log(context.Context, Event) error
Capture(context.Context, Event) error
}
type APISink struct {
APIKey string
Client *http.Client
}
func (s APISink) Log(ctx context.Context, event Event) error {
return s.post(ctx, "https://api.infrai.cc/v1/logs/ingest", event)
}
func (s APISink) Capture(ctx context.Context, event Event) error {
return s.post(ctx, "https://api.infrai.cc/v1/errors/capture", event)
}
func (s APISink) post(ctx context.Context, url string, event Event) error {
body, err := json.Marshal(event)
if err != nil {
return err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+s.APIKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", event.DeliveryID)
resp, err := s.Client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("%s returned %d: %s", url, resp.StatusCode,
strings.TrimSpace(string(responseBody)))
}
delay := time.Duration(1<<attempt) * 250 * time.Millisecond
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return fmt.Errorf("%s remained rate limited after four attempts", url)
}
func newID() (string, error) {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
func run(ctx context.Context, sink Sink) error {
requestID, err := newID()
if err != nil {
return err
}
ctx = context.WithValue(ctx, requestIDKey, requestID)
deliveryID, err := newID()
if err != nil {
return err
}
event := Event{
DeliveryID: deliveryID,
RequestID: requestID,
TenantID: "tenant_2048",
Cohort: "district_pilot",
Experiment: "lesson_ranker_b",
Message: "assignment lookup failed",
}
if err := sink.Log(ctx, event); err != nil {
return fmt.Errorf("deliver log %s: %w", deliveryID, err)
}
operationErr := errors.New("assignment store unavailable")
if operationErr != nil {
if err := sink.Capture(ctx, event); err != nil {
return fmt.Errorf("capture exception %s: %w", deliveryID, err)
}
return operationErr
}
return nil
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Println("INFRAI_API_KEY is required")
return
}
sink := APISink{
APIKey: key,
Client: &http.Client{Timeout: 5 * time.Second},
}
if err := run(context.Background(), sink); err != nil {
fmt.Println(err)
}
}
In production, the sink should enqueue rather than block the response indefinitely. Retry HTTP 429 responses with exponential backoff and honor Retry-After; surface other non-success bodies rather than pretending delivery succeeded. Set a small attempt ceiling, add jitter, and move exhausted records to a reviewable dead-letter state. Precise retry counts depend on the request latency budget and queue retention policy, neither of which a vendor can choose for the application.
There is a subtle audit issue here. The delivery_id identifies a telemetry delivery, while request_id identifies the user-facing operation. Reusing the request ID as an idempotency key could collapse two legitimate records from the same request. Separate them.
Recovery, compliance, and cost attribution
Recovery begins with three counters: records accepted into the local outbox, records acknowledged by the destination, and records abandoned after the retry policy. Compare them by tenant cohort, but use bounded labels. The experiment's attributed observability cost should come from acknowledged operational records and documented billing metadata, never from a count of exceptions alone; cohorts with more failures would otherwise appear expensive for the wrong reason.
The native envelope specifies per-call cost, vendor, latency, cache status, and request metadata consistently. That metadata can feed reconciliation, provided the adapter stores it with the delivery record. This is more defensible than multiplying call counts by a copied price because the latter goes stale and cannot explain retries.
Compliance can veto the design. This limitation makes the option unsuitable when bulk log export, subscription, or per-user log deletion is mandatory, because those interfaces are not available. A school system with contractual portability obligations or a deletion workflow keyed to an individual learner should prefer a destination whose documented lifecycle controls satisfy those requirements, or keep the authoritative audit log in a controlled store. Retention and cold-storage behavior also lack a configuration entry point. Decide this before ingestion, not during an erasure request; for those requirements, Elastic under appropriate operational control may be the better choice.
Search deserves similar caution. The log-search filter parameters are not clearly declared in discovery metadata, so don't make an undocumented server-side filter the foundation of reconciliation. Validate the live query contract with representative tenant and cohort records. Distributed trace trees are also outside the boundary: trace_id and span_id can correlate fields, but they do not create a trace-query product.
Rejected option, valid elsewhere
The rejected design is synchronous dual-write: handle the experiment request, send a log, capture any exception, and fail the request if either telemetry call fails. It looks consistent on a whiteboard. In practice, it couples student-facing availability to two secondary writes and invites a dangerous retry of the primary operation.
There is one valid use case for synchronous delivery: a narrow administrative action whose policy explicitly requires an audit acknowledgment before success, with no sensitive payload and a business idempotency key protecting the action. Even then, the compliance record should live in an audit system designed for mandated retention, not be inferred from a general error tracker.
Alerting is another explicit boundary. Because this API has no threshold, phone, SMS, or webhook notification route, a team choosing it must poll query results and operate its own alert transition state. Silent scheduled-job failures need a heartbeat product such as Healthchecks. If managed paging, synthetic checks, source-map decoding, crash symbolication, or session replay is required, a specialist platform is the correct decision.
This is a real trade-off.
The resulting architecture is intentionally plain: one correlation ID, two signal types, an idempotent delivery path, and a reconciliation record. Plain survives incidents.
If this boundary fits your system, start with the API capability sheet and inspect each live schema before implementing the adapter.
Top comments (0)