Short answer: use one completion event per API request, emitted by Express middleware through Pino after the response finishes, and preserve only the fields needed to join latency, status, agent steps, and cost during an incident. Ship those structured logs asynchronously, but treat metrics as the primary fleet-wide latency signal.
That division of labor matters in a fintech agent loop. A histogram can reveal that latency moved; a bounded request event can explain which agent step, route template, and outcome moved with it. Logging every prompt, response body, header, and intermediate object would make reconstruction easier in theory. In practice, it increases retained bytes, exposes sensitive material, and gives every investigation a larger pile to search. Keep less, on purpose.
The unit of design is therefore not "a log line." It is a reconstruction record with a byte budget, a cardinality budget, and an explicit retention window.
Reconstruct the agent loop backward from the decision
Begin an investigation at the financial decision, then walk backward through the agent step, the model usage, the route outcome, and the deployment that handled it. That sequence defines the join keys worth retaining. Place the middleware early enough to establish correlation context, record a monotonic start point, and register completion handling before downstream route code runs. Emit the request-completion event only after the response outcome is known. In Express, that means the event represents the completed exchange rather than the moment the request arrived. Pino should receive a structured object; don't interpolate the same fields into a prose message that must later be parsed.
A useful event schema is small and deliberate:
| Field | Reconstruction purpose | Cardinality treatment |
|---|---|---|
timestamp |
Orders events on the investigation timeline | Retain as an event attribute, not a metric label |
request_id |
Joins the request to agent-step events | High cardinality; logs only |
trace_id |
Joins logs to a distributed trace when tracing exists | High cardinality; logs only |
route_template |
Groups equivalent API work | Bounded; use /payments/:id, never the raw path |
method |
Distinguishes operation semantics | Bounded |
status_code |
Records the response outcome | Bounded |
duration_ms |
Reconstructs end-to-end request latency | Numeric measurement, not a label |
agent_step |
Identifies the loop phase | Controlled enumeration |
model_class |
Separates approved workload classes | Controlled enumeration |
input_tokens and output_tokens
|
Attributes usage to the completed step | Numeric measurements |
error_type |
Groups expected failure classes | Controlled enumeration; exclude raw messages |
The long paragraph is warranted here because field interactions determine whether the record works. A raw URL such as /accounts/83917/transfers?case=9918 is useful for a single request but disastrous as a grouping key: account and case identifiers create unbounded values, query strings can carry secrets, and the same logical operation fragments into thousands of groups. Store the normalized route template for aggregation. Keep a generated request ID in the log event for exact lookup. If policy permits retaining a business correlation value, make it an opaque, purpose-built identifier rather than an account number. For the agent loop, use a controlled agent_step vocabulary such as classify, retrieve, decide, and respond; accepting arbitrary tool names turns a seemingly harmless field into another cardinality leak.
Small fields compound.
Suppose a completion record averages 900 bytes and the service finishes 120 requests per second. Before indexing overhead, replicas, or compression, that is 900 x 120 x 86,400 = 9,331,200,000 bytes per day. The arithmetic isn't a forecast; actual encoded size and compression depend on the pipeline. It is a review technique. Measure representative serialized events, multiply by observed event volume and retention days, then repeat the calculation for each duplicated destination. I'm not sure what compression ratio your payload mix will achieve, so a sampled production measurement should replace an assumed ratio before capacity approval.
How can structured middleware logs preserve request response latency and status?
OpenTelemetry defines a metric as a measurement captured at runtime, and its metrics model supports aggregations such as sums, gauges, and histograms. That makes a histogram the appropriate primary view for request-duration distributions. The completion log remains the evidence for a particular request. These signals should share bounded dimensions such as service, deployment, route template, method, and status class, while request IDs and trace IDs stay out of metric attributes.
Cardinality is multiplicative. Ten route templates, five status classes, four agent steps, three model classes, and twelve deployment identifiers can produce 7,200 attribute combinations before region or tenant enters the picture. Adding a customer identifier doesn't add one series; it can multiply the existing space by the active customer count. This is why cost review belongs in schema review. The label set is an allocation decision.
Status code is necessary but insufficient for agent work. A 200 can still contain an application-level refusal or a completed fallback path, while a 429 can identify an expected capacity boundary. Record a bounded outcome field whose values are defined by the application contract, then keep the HTTP status as the transport result. Do not copy arbitrary response text into that outcome field. RFC 5424 provides standardized severity semantics for syslog messages, but severity and HTTP status answer different questions; map log levels to operational urgency rather than deriving them mechanically from every status code.
Use clocks carefully. Wall-clock timestamps place records on a timeline, while elapsed duration should come from a monotonic source so a clock correction doesn't create a negative or distorted latency measurement. The middleware should calculate duration once and use the same value in the completion event and the metric observation. That avoids an investigation in which two nearly identical measurements disagree because they were captured at different boundaries.
Make log loss observable before tuning retention
Sampling after an event is emitted can reduce downstream storage, but naive random sampling discards the very requests an incident review needs. A better policy preserves all bounded failure outcomes, security-relevant decisions, and unusually slow requests, then samples ordinary successful completions at a documented rate. The catch is that tail-aware selection needs a latency threshold or later pipeline decision; a purely head-sampled request cannot know its final status or duration. Any such policy needs counters for events accepted, exported, sampled, and dropped, because an empty search result is otherwise indistinguishable from an uneventful request.
Don't use sampled log counts as exact traffic totals. A sampling rate can support an estimate when selection is understood, yet metrics are the cleaner source for request rate, latency distributions, and status totals. Logs answer the narrower reconstruction question: what happened to this request, in what order, under which deployment and policy context?
Retention should follow investigation value rather than habit. Keep dense, searchable completion events for the window in which responders normally investigate; move any longer-lived audit record into a schema and control plane designed for that obligation. The two records may overlap, but they shouldn't be conflated. An operational event optimized for route and latency search is not automatically a compliant financial audit record.
Shipping also needs a failure policy. The request path must not wait for a remote log destination, because telemetry latency would then contaminate the latency being measured. Buffer locally or through an adjacent collector, place a hard bound on memory and disk use, and expose dropped-event counts as metrics. Under sustained pressure, preserve the application and report telemetry loss explicitly. Silent, unbounded buffering is not reliability.
Here is a compact verification pass for an implementation. The shell assumes API_ORIGIN and LOG_SEARCH_ORIGIN identify test environments. It sends one ordinary request and one validation failure with explicit correlation IDs, then queries the internal log-search interface to confirm there is one completion event per request and that both carry the required bounded fields.
curl --fail-with-body \
--request POST \
--header 'content-type: application/json' \
--header 'x-request-id: qc-agent-001' \
--data '{"instruction":"classify transfer risk"}' \
"${API_ORIGIN}/agent-runs"
curl --fail-with-body \
--request POST \
--header 'content-type: application/json' \
--header 'x-request-id: qc-agent-002' \
--data '{}' \
"${API_ORIGIN}/agent-runs"
curl --fail-with-body \
--get \
--data-urlencode 'request_id=qc-agent-001' \
--data-urlencode 'request_id=qc-agent-002' \
"${LOG_SEARCH_ORIGIN}/search"
The check should assert absence too: no authorization header, cookie, raw prompt, response body, account identifier, query string, or stack trace belongs in the normal completion event. A redaction filter is defense in depth, not permission to collect everything first.
Migrate by replaying incident questions
Start in shadow mode: create the completion event, measure its serialized size and field presence, but keep it out of the long-retention index. Exercise successful, rejected, timed-out, and client-disconnected requests in preproduction. Confirm that exactly one completion event is produced for each terminal path and that duration_ms, status_code, route_template, and correlation identifiers agree with the corresponding metric and trace context.
Then enable a small traffic slice for one retention cycle. Review three things before expanding: bytes per completed request, the number of distinct values for every proposed grouping field, and whether an investigator can reconstruct an agent loop without opening payload bodies. A schema that misses a decision point should gain a bounded field. A schema that asks responders to search raw prose should be corrected before scale makes the mistake expensive.
This approach is not suitable when the primary requirement is exact cross-service causal ordering; use distributed tracing for that and correlate the trace ID back to logs. Stick with metrics when the question is only fleet-level latency or error rate. Use a separately governed audit trail when regulation requires immutable business-event retention. Structured completion logs occupy the middle: detailed enough for incident reconstruction, bounded enough to operate deliberately.
No single sampling or retention number is universally correct. Traffic shape, investigation delay, legal obligations, and event size decide it. The durable rule is to calculate retained bytes, constrain cardinality before deployment, and test reconstruction against real failure classes without retaining sensitive payloads.
Top comments (0)