A media notification service cannot attribute cost from a grouped exception count. The operational constraint changes the choice: retries, duplicate queue deliveries, and silent misses make issue totals diverge from work performed. Use three separate ledgers for requested notifications, delivery attempts, and captured exceptions; then choose an error-tracking system by how well it preserves joins among them without turning high-cardinality context into an uncontrolled ingestion bill.
TL;DR: the application database owns notification and attempt accounting. The error tracker owns searchable diagnostic evidence and grouped issues. Join them with opaque identifiers and a W3C trace ID, but price and test the system against raw event volume, retention, and regional data flow rather than the number of visible issues.
I have been paged for missed jobs and duplicate deliveries while running cron and queue infrastructure in production. The useful postmortem lesson was not that one more alert would fix scheduling. It was that three quantities which looked interchangeable on a quiet day separated under failure: requested work, attempted work, and exception evidence.
They are different books.
What should FastAPI and Django test in an error tracking API?
Imagine one publisher requesting a breaking-news notification for email and push. The email attempt succeeds. The push attempt times out, a retry starts, and a duplicate queue delivery reaches another worker. An error tracker might place both timeout exceptions into one issue. That grouping is useful for triage, but the single issue represents neither one customer request nor the actual number of delivery attempts.
The reverse failure is just as important. A worker can stop after an attempt row is created but before exception capture. The notification ledger shows requested work, the attempt ledger shows unfinished work, and the exception system shows nothing. Absence of an exception event is not evidence of successful delivery. Rails and Laravel workers face the same boundary; language and framework conventions do not alter the accounting units once work crosses into a queue.
No exception.
This gives the design its invariant: cost allocation comes from durable business and attempt records; exception events enrich those records for diagnosis. A grouped issue may change shape when a grouping rule changes, so it cannot be the accounting unit.
| Ledger | Stable unit | Answers | Must not be inferred from |
|---|---|---|---|
| Notification | One logical message request | What work did the tenant request? | Exception count |
| Attempt | One execution against one channel | What resources and downstream calls were attempted? | Grouped issue count |
| Exception | One captured failure observation | What failed, where, and with which context? | Missing attempt rows |
This separation also sharpens the buying question. A low-operations service is not merely one with easy middleware. It must let an operator search individual events by the application's opaque join keys, inspect the raw events behind a group, define retention appropriate to the data, and verify the path taken by US or EU data. Those are evaluation criteria, not endorsements.
Put an explicit budget before the capture call
The expensive mistake starts with indiscriminate context. Stack traces and the original error are diagnostic evidence. Tenant IDs, attempt IDs, channel names, and job classes can support allocation and reconstruction. Subscriber addresses, message bodies, tokens, and arbitrary exception text do not belong in searchable attributes. They widen the privacy surface, and volatile values create unbounded cardinality.
I first treated exception messages as convenient issue keys. Retry paths make that assumption brittle: a message containing a request-specific value can fragment one failure class into many groups, while an overly broad key can merge failures that require different runbook actions. The correction is an application-owned, reviewed failure class with low cardinality. Keep the original error for inspection, but do not make its changing prose the accounting key.
The preventative path should be boring. The store decides whether this logical channel delivery may proceed; the sink receives bounded evidence after the durable state transition. Failure to report telemetry never turns a failed delivery into a success.
package delivery
import (
"context"
"errors"
)
type Attempt struct {
ID string
TenantID string
NotificationID string
Channel string
IdempotencyKey string
TraceID string
}
type Store interface {
BeginOnce(context.Context, Attempt) (bool, error)
MarkDelivered(context.Context, string) error
MarkFailed(context.Context, string, string) error
}
type Sender interface {
Send(context.Context, Attempt) error
}
type ErrorSink interface {
Capture(context.Context, error, map[string]string) error
}
func Deliver(ctx context.Context, a Attempt, db Store, sender Sender, sink ErrorSink) error {
created, err := db.BeginOnce(ctx, a)
if err != nil {
return err
}
if !created {
return nil
}
if sendErr := sender.Send(ctx, a); sendErr != nil {
failureClass := "downstream_delivery_failure"
if stateErr := db.MarkFailed(ctx, a.ID, failureClass); stateErr != nil {
return errors.Join(sendErr, stateErr)
}
_ = sink.Capture(ctx, sendErr, map[string]string{
"tenant_id": a.TenantID,
"attempt_id": a.ID,
"channel": a.Channel,
"failure_class": failureClass,
"trace_id": a.TraceID,
})
return sendErr
}
return db.MarkDelivered(ctx, a.ID)
}
The idempotency key and attempt ID serve different purposes. The former stays stable for the logical side effect so duplicate delivery can be rejected. The latter identifies the execution record used in reconciliation. A trace ID is correlation context, not either kind of business identity. W3C Trace Context defines interoperable traceparent and tracestate headers, which is preferable to inventing a framework-specific propagation format when work crosses services.
Framework middleware still has a role. It can capture unhandled request exceptions in FastAPI, Django, Rails, or Laravel, but a notification usually leaves the request process and enters a queue worker. The enqueue path must carry the business keys and trace context explicitly. Otherwise the easiest exceptions to collect are the least useful ones during a delivery incident.
Reconcile cost with a failure matrix
Do not begin the evaluation with dashboard screenshots. Begin with a fixed test tenant and a compact failure matrix: one logical notification, two channels, a retryable downstream failure, a duplicate queue delivery, and a worker exit before exception capture. This is a reconstruction test, not a benchmark, so it does not require invented throughput or latency claims.
For each run, export or query the evidence and compare three independent totals over the same time window. Requested notifications should reconcile to the notification ledger. Execution and downstream resource allocation should reconcile to attempt rows by tenant and channel. Captured exceptions should reconcile only to the subset of attempts that emitted diagnostic evidence. The totals are expected to differ; every difference needs an explanation.
Break it deliberately.
A candidate fails the exercise if operators cannot find an individual event by opaque attempt ID, cannot inspect events inside a group, or cannot determine the documented metering unit. It also fails the organization's regional requirement when the complete data flow cannot be established. A label such as “EU region” is not a data-flow diagram; storage, ingestion, support access, backups, deletion, and applicable subprocessors all need review against the organization's policy and agreement.
Cost comparison then becomes mechanical. Estimate the documented metering unit from observed capture behavior after retries, filtering, sampling, and retention rules. Add the operational work needed to maintain grouping rules, redaction, exports, and reconciliation. Do not convert open-issue counts into tenant cost because a grouping change can alter that number while the underlying workload remains unchanged.
Deployment should follow the same model. Start with a test tenant, verify redaction, and expand capture in controlled steps. Alert separately on overdue attempt rows and rejected telemetry. If a local telemetry buffer is used, it needs a fixed capacity, expiry behavior, and its own saturation signal; application delivery must not depend on an unbounded observability queue.
When is the 3-ledger method too much?
A synchronous internal application with no tenant allocation, asynchronous queue, or external side effect may not need all three ledgers. Structured logs with stable correlation fields and database state can be sufficient. The Twelve-Factor App describes logs as event streams and recommends that applications avoid managing log routing themselves; searchable operation still depends on an external destination and an explicit retention policy.
The limitation is real: this method adds another schema boundary, reconciliation logic, access control, and a runbook that distinguishes silent work failures from captured exceptions. That trade-off is justified when notification cost or customer impact must be attributed across retries and channels. For a small tool where that distinction changes no operational decision, choose structured logs and database state instead. A high-volume pipeline with an established telemetry platform may also choose direct event export and offline allocation rather than add a separate error-tracking service; the cost is slower interactive investigation, while the benefit is one fewer operational dependency.
Small systems should stay small.
The selection rule remains narrow: choose the system whose raw-event search, grouping transparency, retention controls, regional evidence, and documented metering unit preserve the three-ledger reconciliation. Framework support reduces setup time. It does not repair an accounting model that treats an issue as work performed.
Sources
- The Twelve-Factor App, “Logs”: https://12factor.net/logs
- W3C, “Trace Context”: https://www.w3.org/TR/trace-context/
Top comments (0)