For a production logistics agent, error tracking should use one failure envelope at every execution boundary, then attach cost and latency before the boundary reports or rethrows the failure. An HTTP exception filter alone cannot do that job: scheduled route refreshes and queue-driven shipment checks never enter the HTTP pipeline. The operational recommendation is to keep framework adapters thin, make one recorder own normalization and redaction, and preserve each boundary's native retry semantics.
This is a cost-attribution problem as much as an exception-capture problem. If an agent loop calls a model three times, queries a carrier twice, and then fails while persisting a route, a stack trace without tenant_id, workflow_id, attempt, elapsed time, and accumulated usage tells the on-call engineer where execution stopped but not which customer or workflow consumed the budget. Recording only failed calls is also insufficient: the cost was incurred by the whole attempt, including successful steps before the exception.
Don't make the interceptor the architecture. Make it one adapter.
How should NestJS error tracking capture HTTP exceptions, cron jobs, and queue workers?
Treat HTTP requests, cron invocations, and queue deliveries as three separate execution boundaries feeding the same internal recorder. The HTTP interceptor can establish timing and correlation context around controller execution, while the exception filter translates an uncaught HTTP-path failure into the shared envelope and preserves the response contract. A cron wrapper creates a fresh execution context because there is no inbound request. A queue-worker wrapper starts from message metadata, records the current delivery attempt, and rethrows so the queue system, rather than the tracking layer, decides whether to retry or dead-letter the message.
The envelope should separate bounded dimensions from searchable detail. Fields such as boundary, operation, outcome, and a low-cardinality error class are useful for SLO aggregation. Shipment IDs, workflow IDs, carrier responses, and stack traces belong in event detail, not metric labels. A tenant identifier may be necessary for chargeback, but exposing raw customer data to every telemetry sink is not; use an approved internal identifier and apply the same redaction policy before any exporter sees the event.
I use one test to reject designs quickly: can the same logical workflow be reconstructed when its initial HTTP request returns before the queue work begins? If correlation depends on request-local state, the answer is no. Carry an explicit workflow identifier into the job payload, establish a new span or execution context in the consumer, and retain the causal link. Don't pretend one process-local context crosses an asynchronous handoff.
The table is the buy-versus-build decision in miniature:
| Concern | Shared code owns | Boundary adapter owns | Platform or runtime owns |
|---|---|---|---|
| Error normalization | Class, safe message, fingerprint | Boundary name and operation | Stack construction |
| Cost attribution | Accumulated usage and attribution keys | Initial context extraction | Provider usage response |
| Retry behavior | Record attempt and outcome | Rethrow or translate correctly | Backoff and dead-letter policy |
| Latency | Common monotonic duration field | Start and finish around execution | Clock |
| Redaction | Allowlist and masking rules | Raw input selection | Sink access controls |
The catch is ownership. A managed tracker can reduce exporter maintenance, but it cannot infer the right business attribution keys or decide which queue attempts count against an SLO. A self-hosted pipeline offers control over retention and routing, yet it transfers capacity planning, upgrades, and an on-call surface to the platform team. Neither choice removes the need for a stable application envelope.
Build the recorder before its adapters
The safe implementation starts with a framework-neutral contract. The following Go model is intentionally small: every adapter can call it, and exporter choice stays outside request, scheduler, and worker code. In a NestJS codebase the concrete adapters will be TypeScript, but this reference makes the ownership boundaries unambiguous without binding the design to a package API.
package failure
import (
"context"
"errors"
"time"
)
type Boundary string
const (
BoundaryHTTP Boundary = "http"
BoundaryCron Boundary = "cron"
BoundaryQueue Boundary = "queue"
)
type Usage struct {
InputUnits int64
OutputUnits int64
}
type Event struct {
Boundary Boundary
Operation string
WorkflowID string
TenantID string
Attempt int
Duration time.Duration
Usage Usage
Err error
}
type Recorder interface {
Capture(context.Context, Event) error
}
type Work func(context.Context) (Usage, error)
func Observe(
ctx context.Context,
recorder Recorder,
event Event,
work Work,
) (Usage, error) {
started := time.Now()
usage, workErr := work(ctx)
if workErr == nil {
return usage, nil
}
event.Duration = time.Since(started)
event.Usage = usage
event.Err = workErr
captureErr := recorder.Capture(ctx, event)
if captureErr != nil {
return usage, errors.Join(workErr, captureErr)
}
return usage, workErr
}
There is a deliberate limitation here. Observe records failures, while a separate completion metric should record every attempt's duration and usage; forcing both jobs into a single error API tends to create an event stream that cannot answer denominator questions. It also returns the original work error, joined with a capture error when necessary, rather than swallowing it. In the HTTP adapter that error proceeds to the exception filter. In the queue adapter it returns to the consumer runtime. In the cron adapter it reaches the scheduler's failure policy.
For agent loops, update the local Usage accumulator after each external call whose authoritative response contains usage. If a call fails before returning usage, don't invent a number. Record that attribution is incomplete and reconcile it from the provider's authoritative records if those records are available. I'm not sure a universal sampling threshold exists here; event volume, tenant skew, retention, and the financial tolerance for unattributed usage determine it. What is universal is the failure mode: sampling away rare stack traces may be acceptable, while sampling away billing evidence can make chargeback mathematically impossible.
Keep metric labels bounded. A dashboard grouped by boundary and operation supports capacity planning; a label per shipment creates a series per shipment and turns customer growth into telemetry growth. Detailed identifiers can remain on traces or events where indexed retention is an explicit decision.
Deploy capture without changing failure semantics
Instrumentation changes can break production behavior even when the recorder itself is correct. A filter that converts every exception into the same HTTP status damages client contracts. A queue wrapper that acknowledges after capture, rather than after successful work, changes delivery semantics. A cron wrapper that catches and suppresses an error makes a failed route-planning cycle look healthy. Those are control-flow defects, not observability gaps.
Ship the adapters behind an operational feature toggle and separate exposure from code deployment. Martin Fowler's feature-toggle guidance distinguishes toggle categories by longevity and dynamism; an operations toggle fits this rollout because the platform team needs a quick control during deployment, not a permanent branch in business logic. The flag should choose whether the new capture path runs. It should not decide whether application errors propagate.
A staged rollout can begin with one low-risk operation in each boundary, compare old and new event counts, then expand by operation. The exact slice size depends on traffic distribution and error rarity, so your mileage may vary. Capacity planning comes first: estimate attempts per second, expected failure-event bytes, retention, and exporter throughput, then leave headroom for a correlated carrier outage that raises failures across many tenants at once. Normal-day averages are a poor sizing input for an error pipeline.
Define two SLOs before enabling broadly. One covers application work, such as the fraction of route-agent attempts completing within the latency objective. The other covers telemetry delivery, such as the fraction of accepted failure envelopes reaching durable storage within its objective. Keep their error budgets separate. A telemetry miss is serious, but it must not be silently counted as a shipment-processing failure or allowed to mutate the original retry decision.
Short version: observe control flow; don't own it.
Verify attribution and keep rollback boring
Verification needs a boundary-by-boundary matrix, not a single controller test. Inject a known failure before any metered call, after one metered call, and after a queued handoff. Assert that exactly one terminal failure event appears for each attempt, that usage reflects only completed calls with authoritative usage, and that the workflow identifier links the HTTP initiation to later queue work. Then repeat with capture disabled and confirm status codes, thrown errors, retry counts, and acknowledgements remain identical.
Use synthetic tenant and workflow values in these tests. Check the negative space too: request bodies, prompts, credentials, shipment addresses, and raw carrier payloads should not appear unless the data policy explicitly permits them. A redaction test that searches serialized events for forbidden keys catches more than a code review usually will.
For queue workers, test at least first delivery, retry, and terminal dead-letter behavior according to the queue policy already in force. Cost belongs to an attempt and may also need a workflow total; store enough identity to aggregate both without treating a retry as a new customer workflow. For cron jobs, overlap is the capacity trap: if the scheduler permits concurrent executions, include an invocation identifier and verify that two runs cannot overwrite each other's attribution. For HTTP, assert the existing exception class and response status survive capture.
Retries count.
Rollback should be one flag change that stops new capture calls while leaving application execution untouched. Preserve the envelope schema and the tests after rollback; deleting them makes the next rollout another first attempt. Remove the operational toggle once the new path has met its telemetry SLO for a full review window, because a permanent branch doubles the states the on-call engineer must reason about.
This method is not suitable when the immediate requirement is full continuous profiling, log storage, or provider-side billing reconciliation; those are adjacent systems with different data and capacity models. Stick with the existing specialized pipeline for those jobs, and integrate through stable correlation and attribution fields. The error recorder should remain narrow enough that an exporter replacement does not require editing controllers, scheduled tasks, and consumers together.
Top comments (0)