A nightly health-data pipeline changes the usual error-tracking decision: the scarce resource is not another alert, but enough trustworthy context to distinguish one broken transformation from ten thousand repetitions of it. Choose server-side grouping from a conservative fingerprint, preserve the original stack trace, and attach release and environment as context rather than letting either field create a new issue by default.
Short answer: group repeated failures by stable exception identity, keep deployment context searchable, and sample only after the first complete event for each group has been retained.
This is an architecture decision, not an inbox preference. A healthtech SaaS app may process many records in one scheduled run, so one defect can emit a line per rejected record. Counting those lines as separate incidents inflates storage and pages the team without adding diagnostic signal. Collapsing too aggressively is just as damaging: two failures with different causes can look like one healthy, familiar group. The decision must therefore optimize signal quality versus noise, with cardinality and retained bytes treated as design constraints.
Decision record: invariants and failure boundaries
The grouping key should represent the failure mechanism, not the affected record. A useful starting fingerprint contains a normalized exception type, the application-owned top stack frame, and a stable operation name such as normalize_lab_result. It excludes patient identifiers, request identifiers, timestamps, raw messages containing values, and the nightly run ID. Those excluded fields remain event attributes when policy permits, but they don't belong in identity.
Three invariants follow. First, the same code defect processing different records should converge on one group. Second, two defects at different application-owned frames should remain separable even when their outer exception text matches. Third, changing an environment or release should enrich the timeline of a group instead of silently multiplying its issue count. These rules keep the number of groups tied roughly to distinct failure mechanisms rather than input volume.
The failure boundaries matter more than the happy path. If symbol or source information is absent, retain the raw stack and mark the event as lacking symbolization; don't replace the fingerprint with the full message. If an exception is wrapped, preserve the causal chain and select the deepest application-owned cause that remains stable across equivalent executions. If no application frame exists, fall back to a deliberately coarse exception-type-plus-operation key and route the event for review. A fallback is allowed to merge; it must not explode cardinality.
OpenTelemetry's logs model gives this design a neutral transport vocabulary. Logs can carry a timestamp, observed timestamp, severity, body, resource context, attributes, and trace or span identifiers where those identifiers exist. Resource attributes can describe the service and deployment context once, while event attributes describe the particular failure. That distinction is useful here: service.name and an environment label belong to the emitting resource; operation, exception details, and the computed fingerprint describe the event.
Privacy is a separate boundary. A grouping system does not need raw health data to identify a code failure. Scrub or omit sensitive values before export, and test that rule with representative payload shapes. Hashing an unstable or sensitive value and putting the hash in the fingerprint still creates one group per value; it hides the text but does not solve cardinality.
Small key, rich event.
How should a SaaS app connect error grouping, fingerprinting, stack traces, releases, and environments?
Treat the issue as a durable hypothesis: these events share one fix. The fingerprint encodes that hypothesis. A stack trace supplies the strongest code-location evidence, while release and environment answer when and where the hypothesis holds. Error tracking then becomes a sequence of tests against the group rather than a stream of unrelated log lines.
Suppose a nightly import processes 80,000 records and 2,400 reach the same parser branch with an unexpected unit. Those numbers are an illustrative workload, not a benchmark. If the raw unit value or record ID enters the fingerprint, the system can create thousands of groups. A fingerprint containing only ParseError may collapse unrelated parser defects together. A more disciplined key such as ParseError | normalize_lab_result | parser.go:184 yields one candidate group for that code location, while sanitized unit category, release, environment, run ID, and record outcome remain searchable attributes. The first full event carries the complete stack and causal chain; subsequent matching events can increment counts or be sampled according to a declared policy.
This separation also makes release analysis honest. A release is not proof of causation. It is a boundary that lets an operator ask whether a group first appeared, disappeared, or changed frequency after deployment. Environment works the same way. Production, staging, and a replay environment may all observe the same underlying defect, yet their volume and data shape differ. Keeping those labels outside the default fingerprint permits comparison without turning one defect into three inbox items.
There is a catch: a stable fingerprint can conceal a behavioral split introduced by a new release. Preserve per-release counts and a small number of representative full events, then split a group manually or through a reviewed rule when stack evidence or remediation differs. The threshold for that split cannot be universal; it depends on traffic, deploy cadence, and the cost of a false merge. Your mileage may vary, and a week of labeled triage decisions is better evidence than a fashionable default.
Compare grouping choices by signal, noise, and cost
The table is an architecture comparison, not a vendor scorecard.
| Grouping strategy | Signal quality | Cardinality behavior | Retention consequence | Appropriate use |
|---|---|---|---|---|
| Full message | High only when messages are already normalized | Unbounded when values appear in text | Repeated near-duplicates consume storage | Controlled internal errors with fixed messages |
| Exception type only | Low for applications with broad wrapper types | Very low | Cheap, but representative events mix unrelated causes | Emergency fallback when stacks are absent |
| Type plus application frame and operation | Usually strong for code defects | Bounded by code locations and named operations | Full exemplars can be retained per group; repeats can be counted | Default for the nightly pipeline |
| Release or environment inside the key | Makes deployment slices visually obvious | Multiplies groups across deployments and stages | Repeats stack history and metadata | Temporary isolation during a migration, with an expiry rule |
The default wins because its cardinality has an understandable upper bound. It isn't perfect. Generated code, asynchronous boundaries, wrapper functions, and source-map changes can move the apparent top frame without changing the defect. Normalize known framework frames and prefer the first application-owned frame, but keep the unmodified stack as evidence. Never discard the input needed to revise a bad grouping rule.
Retention math should be explicit even when prices are not. Let G be distinct groups, E the retained full exemplars per group, B the average bytes per full event, and C the bytes for one compact repeat counter. The approximate stored volume for one retention window is G × E × B + repeats × C, plus index overhead. The equation exposes the lever: sampling duplicate events reduces the second-order stream only after grouping is trustworthy. Sampling before fingerprint computation can erase the first occurrence of a rare, actionable defect.
Don't sample the evidence first.
For the same reason, avoid indexing every attribute. High-cardinality values may be useful in short-lived forensic storage while being poor global index fields. Decide separately whether a field is collected, retained, indexed, displayed, or admitted into a fingerprint. Those are five different cost and risk decisions, though many beginner setups accidentally collapse them into one checkbox.
Critical path: send one structured exception event
The ingestion boundary should accept a structured record and return an acknowledgement only after validating required fields. The following pseudonymous endpoint illustrates the contract with a single curl command; it is not a route for any named product.
curl --fail-with-body \
--request POST \
--header 'content-type: application/json' \
--data '{
"timestamp": "2026-08-16T02:14:31Z",
"severity_text": "ERROR",
"body": "Lab result normalization failed",
"resource": {
"service.name": "nightly-results-normalizer",
"deployment.environment.name": "production",
"service.version": "2026.08.16.1"
},
"attributes": {
"error.type": "ParseError",
"error.operation": "normalize_lab_result",
"error.fingerprint": "ParseError|normalize_lab_result|parser.go:184",
"error.stack": "ParseError: invalid unit category\n at normalize_lab_result (parser.go:184)",
"pipeline.run_id": "run_example_481"
}
}' \
https://telemetry.example.test/v1/logs/ingest
The example deliberately omits a patient identifier and the rejected value. The run ID is correlation context, never a grouping component. In a real deployment, schema validation should reject missing fingerprint inputs, privacy tests should exercise nested and malformed records, and a canary should verify that one synthetic exception appears in the intended environment and release slice before the new pipeline version handles production data.
Operationally, the collector needs a bounded buffer, backpressure behavior, and a declared response to export failure. The application should not turn telemetry delivery into failure of the health-data job itself. Preserve a local count of dropped or rejected telemetry, alert on that count through an independent path, and make retry limits finite. This is a trade-off: decoupling protects the primary workload, but it means the error tracker cannot be treated as a complete audit ledger. Compliance evidence and business processing records need their own durable system of record.
Testing should focus on invariants. Feed the grouper two events with different run IDs and confirm one fingerprint; move the application frame and confirm a second fingerprint; change only the release and confirm one group with two release facets; remove the stack and confirm the coarse fallback. Then add a sensitive nested value and verify that it is absent before export. These tests catch the expensive mistakes before retention fills with unique labels.
Rejected option: retaining every raw failure as a separate issue
The rejected design creates an issue for every failed record and keeps the complete payload for the full retention period. It is not suitable for a repetitive nightly SaaS pipeline: issue count tracks batch size, triage repeats the same decision, and sensitive data exposure expands without improving the fix hypothesis. Stick with per-event issues when each event represents a genuinely independent workflow that requires individual adjudication, such as a small manual-review queue where resolution belongs to the record rather than the code defect.
There is another valid exception. During a short, controlled schema migration, temporarily placing a migration cohort in the grouping key can separate remediation paths. Document the expiry condition before deployment. Without one, a diagnostic dimension tends to become permanent cardinality.
The final decision is intentionally narrow: stable code-based fingerprints for identity; stack traces for evidence; releases and environments for facets; complete exemplars before duplicate sampling; and explicit privacy, buffering, and retention boundaries. It keeps the nightly pipeline searchable without pretending every log line deserves to become an incident.
Top comments (0)