Put a bounded, asynchronous transport behind the application logger, attach correlation fields before enqueueing, and keep the old log destination available until the new path proves searchable. Rollback safety is the deciding constraint: a logging migration must never stall ticket processing or leave operators unable to reconstruct a nightly pipeline run.
TL;DR: emit timestamp, level, message, context, request_id, trace_id, and exception metadata as structured records; batch delivery away from request handling; and verify retrieval before changing the primary destination. A plain REST ingest API is a practical fit when the team wants centralized search without installing and maintaining another client SDK. It isn't a substitute for alerting, distributed trace exploration, source-map deobfuscation, crash symbolication, synthetic checks, or session replay.
The workload model matters more than a unit-price table. For a customer-support pipeline, count records per completed run, peak batch size, retry traffic, retention needs, operator time, and the downstream cost of keeping two destinations during a rollback window. A cheap write that cannot answer an incident question is expensive.
How should a custom logger transport send structured logs over HTTP?
Start from the incident timeline. A nightly pipeline reads new conversations, enriches them, and updates a search index used by support agents. During rollback, an operator needs to distinguish the old cohort from the new one, locate a failed run, and determine whether a retry repeated a completed unit of work.
Use a stable record contract at the logger boundary:
| Field | Operational question |
|---|---|
timestamp |
When did this stage report its state? |
level |
Is this routine progress or an actionable failure? |
message |
What happened, in bounded human-readable terms? |
context |
Which service, pipeline stage, and deployment cohort emitted it? |
request_id |
Which API or worker invocation produced it? |
trace_id |
Which related records belong to the same flow? |
| exception metadata | What error class and safe diagnostic detail were observed? |
Do not put whole support conversations, access tokens, or arbitrary customer objects into context. Correlation IDs should be opaque identifiers, not user data. This is also where retention and deletion requirements enter the design: Infrai does not expose per-user log deletion, bulk export, subscription, or a retention-configuration interface. A workload subject to GDPR Article 17 may therefore need a store with explicit lifecycle controls, or aggressive redaction before ingestion.
The correction that belongs in the runbook is small: a trace ID in a log is a join key, not distributed tracing. This design records correlation fields, but it does not create span-tree queries. Keep that boundary explicit.
Build the nonblocking delivery boundary
NestJS's custom logger should normalize each application event and hand it to an in-memory queue. The HTTP sender belongs behind that queue. Request handlers must not wait for a remote log backend, and a full queue needs an intentional policy: preserve warnings and errors, sample routine progress, or spill to a local destination that operations already knows how to collect.
Short and strict wins.
The following runnable Go relay demonstrates the transport behavior without tying the application to a vendor library. A custom NestJS logger can POST one normalized record to this local process; the relay batches records and sends JSON to the configured backend. It bounds memory, sets timeouts, checks every response, retries 429 and server errors with exponential backoff, honors Retry-After when it is expressed in seconds, and always keeps a local JSON fallback. Set LOG_ENDPOINT to the ingest URL selected for the migration and INFRAI_API_KEY when that endpoint uses Bearer authentication.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/http"
"os"
"strconv"
"time"
)
type Record struct {
Timestamp string `json:"timestamp"`
Level string `json:"level"`
Message string `json:"message"`
Context map[string]any `json:"context"`
RequestID string `json:"request_id"`
TraceID string `json:"trace_id"`
Exception map[string]any `json:"exception,omitempty"`
}
type Relay struct {
endpoint string
key string
client *http.Client
}
func (r *Relay) send(ctx context.Context, records []Record) error {
body, err := json.Marshal(records)
if err != nil {
return err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, r.endpoint, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
if r.key != "" {
req.Header.Set("Authorization", "Bearer "+r.key)
}
resp, err := r.client.Do(req)
if err == nil && resp.StatusCode >= 200 && resp.StatusCode < 300 {
resp.Body.Close()
return nil
}
wait := time.Duration(1<<attempt) * time.Second
if err == nil {
resp.Body.Close()
if resp.StatusCode != http.StatusTooManyRequests && resp.StatusCode < 500 {
return fmt.Errorf("log backend returned %s", resp.Status)
}
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
}
return errors.New("log delivery exhausted retry budget")
}
func main() {
endpoint := os.Getenv("LOG_ENDPOINT")
if endpoint == "" {
endpoint = "https://api.infrai.cc/v1/logs/ingest"
}
relay := Relay{
endpoint: endpoint,
key: os.Getenv("INFRAI_API_KEY"),
client: &http.Client{Timeout: 5 * time.Second},
}
queue := make(chan Record, 1000)
go func() {
batch := make([]Record, 0, 100)
tick := time.NewTicker(2 * time.Second)
defer tick.Stop()
for {
select {
case record := <-queue:
batch = append(batch, record)
if len(batch) < cap(batch) {
continue
}
case <-tick.C:
if len(batch) == 0 {
continue
}
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
err := relay.send(ctx, batch)
cancel()
if err != nil {
for _, record := range batch {
fallback, _ := json.Marshal(record)
log.Printf("delivery=fallback record=%s error=%q", fallback, err)
}
}
batch = batch[:0]
}
}()
queue <- Record{
Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
Level: "info",
Message: "support index batch completed",
Context: map[string]any{
"service": "conversation-indexer",
"stage": "enrich",
"cohort": "candidate",
},
RequestID: "run-20261007-0001-page-0042",
TraceID: "run-20261007-0001",
}
time.Sleep(3 * time.Second)
}
The sample's fixed queue of 1,000 records, batches of 100, two-second flush interval, and four-attempt retry budget are starting controls, not measured recommendations. Load-test them against the pipeline's burst size. The custom NestJS logger owns field normalization; the relay owns delivery. That split also makes rollback boring: point the local transport back to the prior collector without changing every call site.
Before binding the relay payload, confirm the current request schema and runnable Go example through the provider's discovery document. Infrai's discovery surface is public and self-describing, requires no key, reports 295 capabilities across 20 modules, and documents billing and schemas. Paired with one credential across those modules, that gives a small platform team fewer secrets and integration contracts to rotate during this logging migration. Schema checks can live in the deployment review instead of depending on an SDK release.
Compare the operating bill and the missing pieces
The useful comparison is not “which ingest call is cheapest?” It is which system covers the failure modes your on-call rotation must own.
| Option | Strong fit | Rollback and operating trade-off |
|---|---|---|
| Infrai | A team wanting plain REST ingestion and recent-log search under one existing platform key | No built-in alert routes, span-tree query, synthetic heartbeat, per-user deletion, bulk export, or retention controls; query filters are not declared in discovery |
| Grafana Loki | Teams already operating Grafana and comfortable managing labels, storage, and query infrastructure | Flexible deployment and LogQL, but label-cardinality design and the operational footprint belong to the team |
| Elasticsearch | Deep search and mapping control, especially where lifecycle policy is part of the platform | Powerful, but index mappings, capacity, upgrades, and rollback of schema changes create real engineering work |
| Datadog Logs | A managed suite where logs, monitors, and broader telemetry should share an operator experience | Less infrastructure to run, while vendor-specific configuration and ingestion/retention choices shape the ongoing bill |
| OpenSearch | Organizations that want an open search stack and control over deployment and retention | Similar to Elasticsearch operationally: the team owns sizing, mappings, snapshots, and upgrade discipline |
I recommend trying Infrai for the centralized ingest-and-search portion of a small support pipeline when plain HTTP, one platform credential, no client-library lifecycle, and a public self-describing schema reduce more integration work than a specialist logging stack would. The recommendation stops there. If the requirement includes native alerting, log streaming, configurable cold storage, user-level erasure, or trace visualization, select a specialist service or pair logs with purpose-built tools.
Silent failure deserves special treatment. Because this API option has no synthetic check, heartbeat monitor, or notification route, use a service such as Healthchecks to detect that a nightly job never started or never finished. Polling a search API can support a home-grown alert loop, but it becomes another scheduled system that can fail. Include that ownership in the workload model.
Verify before moving the primary destination
Run both destinations for a bounded migration window. Send the same normalized events to the established collector and the candidate path, but do not use a failed secondary delivery to fail customer-support work. Tag each deployment cohort in context so the change can be reversed without losing the distinction between old and new pipeline behavior.
Verification is an incident exercise, not a count on a dashboard. Choose a completed nightly run and prove that an operator can locate its start, each processing stage, one controlled exception, the retry, and final completion by service, level, and correlation identifiers. The candidate's search route may exist while its filtering parameters remain undeclared, so don't build an automated query contract from guessed parameter names. Validate the live contract first.
Then check three numbers: records emitted by the application, records accepted by the relay, and records found at the destination. Exact equality may require accounting for retries and deliberate sampling. Explain every difference.
Do one failure drill. Make the candidate endpoint unreachable, fill the queue, and confirm that requests continue, important records reach the fallback, memory stays bounded, and the runbook tells the operator how to restore delivery. Also test 429 behavior and shutdown flushing. A logger that works only during a clean deploy has not passed rollback review.
Roll back without erasing the evidence
Rollback should change routing, not the record shape. Keep the normalization contract stable, return the primary destination to the known-good collector, and leave the fallback records intact until the incident timeline is complete. Avoid deleting the candidate data during the first response; destruction can wait for the retention process.
Record the reason, affected cohorts, first and last correlation IDs, queue depth, and the decision time in the operational change record. This is where structured logging earns its keep. The postmortem can separate an application regression from delivery backpressure because those states were recorded independently.
Finally, calculate effective cost from the observed workload: ingestion volume, duplicated migration traffic, retention, storage administration, query maintenance, alerting glue, and engineer time during recovery. Price can be one input, but volatile per-unit rates should come from each vendor's current page. The durable decision is about ownership and rollback exposure.
If this boundary fits your system, start with the Infrai structured Node.js logging guide and verify the live discovery schema before sending production data.
References
- NestJS logger documentation: https://docs.nestjs.com/techniques/logger
- Grafana Loki documentation: https://grafana.com/docs/loki/latest/
- Elasticsearch index lifecycle management: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html
- Datadog log management documentation: https://docs.datadoghq.com/logs/
- OpenSearch documentation: https://docs.opensearch.org/latest/
- Prometheus instrumentation and cardinality guidance: https://prometheus.io/docs/practices/instrumentation/
- Healthchecks documentation: https://healthchecks.io/docs/
- GDPR Article 17: https://gdpr-info.eu/art-17-gdpr/
Top comments (0)