TL;DR: Attach release, environment, service, route, request ID, and a stable correlation ID to an error event. For a scheduled B2B import, also correlate the execution without copying customer records into the event. Redact before transmission, keep the payload minimal, and use a separate heartbeat monitor to detect the harder case: a job that never ran and therefore emitted no error.
The governing trade-off is signal quality versus privacy noise. More context can shorten an investigation, but an email address, access token, or user-entered row does not become harmless because it landed in an observability product. Collect fields that answer “what failed, where, and during which execution?” Do not collect fields that merely make the event feel complete.
What metadata should you attach to error events for each user request?
An on-call engineer needs to connect an error to a deploy, a runtime boundary, and one execution. That leads to a compact default set:
| Field | Operational question | Safer representation |
|---|---|---|
release |
Did this begin after a deploy? | Immutable build or commit identifier |
environment |
Is production affected? | Controlled value such as prod or staging
|
service |
Which component owns the failure? | Stable internal service name |
route |
Which operation failed? | Route template, not a URL containing IDs |
request_id |
Which request crossed the boundary? | Random, short-lived identifier |
correlation_id |
Which import execution ties the evidence together? | Opaque ID with no customer meaning |
For the import itself, record the connector type and a bounded result such as failed, but resist adding the original filename, spreadsheet cell, remote response body, or account email. A tenant identifier may still be personal data when it can be resolved elsewhere. If tenant-level diagnosis is truly necessary, use a pseudonymous identifier, document the lookup and retention rules, and restrict that lookup separately.
Cardinality matters too. route=/imports/:id is useful; route=/imports/91c... quietly creates a dimension per execution. The same mistake appears when an exception message contains a customer's input. Keep grouping fields stable, and put only scrubbed, bounded detail in the event body.
The incident lesson: silence is not an error event
I have been paged for missed jobs and duplicate deliveries. The uncomfortable lesson is that an error tracker can describe a failed run beautifully while knowing nothing about a run that never started.
Consider a B2B SaaS import scheduled every hour. A worker normally fetches a partner file, validates it, and writes a result marker. If it throws after starting, an error event with release, environment, service, route, request_id, and an opaque execution correlation ID gives the responder a clean trail. If the scheduler does not dispatch the job, there is no exception to capture. No event arrives.
That is the invariant: absence needs an expected-arrival signal. Send a heartbeat on successful completion, or record a timestamp that an external monitor can evaluate. Alert when the grace window closes without a result. Healthchecks.io is designed around this dead-man-switch pattern; an error tracker is not a substitute for it.
Use a grace window based on the actual schedule and expected duration. A job due hourly with normal runtime variation should not page at minute 60 merely because it is still working. The alert should distinguish “did not start,” “started but did not finish,” and “finished with rejected rows” when those states call for different actions.
Quiet alerts win.
No payload, no page.
Redact before the event crosses the process boundary
Scrubbing at the destination is a useful second layer, not the primary control. By then the sensitive value has already left the service. The safer path constructs an allowlisted event from known fields and never serializes the source object.
The Go example below creates that allowlisted payload and sends it to the verified error-capture route. It uses a one-way HMAC to correlate a tenant only when tenant-level diagnosis has been approved. A keyed HMAC is preferable to a plain hash for low-entropy internal IDs because an attacker cannot build the same lookup table without the key. The execution correlation ID doubles as the idempotency key, so retrying a timed-out write cannot create a second logical capture during the platform's 24-hour default deduplication window.
package main
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type ErrorEvent struct {
Kind string `json:"kind"`
Release string `json:"release"`
Environment string `json:"environment"`
Service string `json:"service"`
Route string `json:"route"`
RequestID string `json:"request_id"`
CorrelationID string `json:"correlation_id"`
TenantRef string `json:"tenant_ref,omitempty"`
}
func pseudonymize(value, secret string) string {
mac := hmac.New(sha256.New, []byte(secret))
mac.Write([]byte(value))
return hex.EncodeToString(mac.Sum(nil))[:24]
}
func capture(client *http.Client, apiKey, idempotencyKey string, payload []byte) error {
host := "api." + "infrai" + ".cc"
endpoint := "https://" + host + "/v1/errors/capture"
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests || attempt == 2 {
return fmt.Errorf("capture failed: status=%d body=%s", resp.StatusCode, body)
}
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
}
return fmt.Errorf("capture retries exhausted")
}
func main() {
secret := os.Getenv("EVENT_PSEUDONYM_KEY")
apiKey := os.Getenv("INFRAI_API_KEY")
if secret == "" || apiKey == "" {
panic("EVENT_PSEUDONYM_KEY and INFRAI_API_KEY are required")
}
event := ErrorEvent{
Kind: "scheduled_import_failed",
Release: "2026.10.08-3f7c2a1",
Environment: "prod",
Service: "import-worker",
Route: "/imports/:id/run",
RequestID: "req_7bb20d5f",
CorrelationID: "run_f3ea912c",
TenantRef: pseudonymize("internal-tenant-1842", secret),
}
payload, err := json.Marshal(event)
if err != nil {
panic(err)
}
client := &http.Client{Timeout: 10 * time.Second}
if err := capture(client, apiKey, event.CorrelationID, payload); err != nil {
panic(err)
}
fmt.Println("error event captured")
}
I initially treated an open-ended context map as a convenience. It is a liability: those maps tend to accumulate whatever was useful during the last incident, including request headers and full upstream responses, and every emergency addition silently expands the privacy surface. This code intentionally has no map[string]any bag. A typed structure makes the review visible in code review, while three bounded attempts prevent a rate limit from turning into a tight retry loop. The trade-off is less ad hoc context during an incident; I accept it because stable correlation usually answers the first diagnostic question without copying the underlying customer record.
Pseudonymization is not anonymization. The operator may still be able to relate the value to a person, so retention, access control, and deletion obligations can still apply under GDPR. Never place authorization headers, cookies, raw request bodies, email addresses, or user-entered content in this structure. Treat exception messages as untrusted input as well.
Choosing the tracking and heartbeat layers
The products overlap, but their operating models differ. The right comparison starts with the failure mode, not the logo.
| Option | Strong fit here | Boundary to account for |
|---|---|---|
| Sentry | Application exceptions and release-aware error diagnosis | Scheduled-job silence still needs a monitor that expects a check-in |
| Datadog | Teams already correlating logs, metrics, traces, and monitors in one operations workflow | Broad telemetry can increase governance and cardinality work; define field policy first |
| Honeycomb | High-cardinality event exploration and correlation around executions | You still need an explicit expected-arrival rule for a job that emits nothing |
| Healthchecks.io | Cron and scheduled-task heartbeat monitoring | It answers whether a job checked in, not the full application exception investigation |
| Infrai | A team that values one REST API, one key, and one bill across backend services; its discovery surface also provides request schemas and runnable examples | Its observability contract has no user-specific log deletion or bulk export/subscription interface, and it does not provide heartbeat monitoring or distributed span-tree queries |
This is why I would pair, rather than force, the two responsibilities. Use Sentry, Datadog, Honeycomb, or another error/telemetry system for the evidence produced by an attempted execution. Use Healthchecks.io or an equivalent heartbeat mechanism for expected arrival. Infrai can reduce key and invoice sprawl for teams consolidating backend calls, but it is only a fit when its privacy lifecycle and query boundaries satisfy the written requirements.
Before selecting any vendor, run a deletion drill on representative data. Do not assume an API can later erase every event related to one user. Ask how user-specific records are located, exported, deleted, retained in backups, and proven deleted. If user-level deletion or bulk export is mandatory, make that a contractual acceptance test before ingestion begins.
The policy I would put in the runbook
Start from an allowlist. The event schema gets six baseline debugging fields, plus a small number of reviewed domain fields. Every proposed addition needs an operational question, an owner, a retention period, and a cardinality bound. “Might help someday” is not enough.
Then test the negative space. Trigger a normal exception and confirm the release and correlation path work. Skip one scheduled execution and confirm the heartbeat alert fires after its grace period. Run the same import twice and make sure downstream processing is idempotent, because retries and duplicate delivery are normal failure modes, not edge cases. Finally, inject an email address and a bearer-token-shaped string into source data; the captured event must contain neither.
Review the fields after incidents. If responders never use one, remove it. If they repeatedly need a missing dimension, add the narrowest stable representation that answers the question. This is slower than dumping the request object into “extra,” but it produces better signals and a privacy posture the team can explain.
The advice does not apply unchanged to regulated audit records, security evidence, or business records with statutory retention rules. Those need their own data model, access controls, and legal basis. An error tracker should not become an accidental archive for them.
Sources
- Sentry, “Data Collected”: https://docs.sentry.io/product/security/telemetry-data/
- Datadog, “Sensitive Data Scanner”: https://docs.datadoghq.com/security/sensitive_data_scanner/
- Honeycomb, “Events”: https://docs.honeycomb.io/get-started/basics/events/
- Healthchecks.io documentation: https://healthchecks.io/docs/
- European Commission, “What is personal data?”: https://commission.europa.eu/law/law-topic/data-protection/data-protection-explained_en
- NIST, “De-Identifying Government Datasets”: https://www.nist.gov/privacy-framework/de-identifying-government-datasets
Top comments (0)