DEV Community

YorkHolloway3257
YorkHolloway3257

Posted on

Notification Delivery Observability: 3 Checks for Attributing Failed Backend Tasks

Choose backend task monitoring tools and health check APIs by testing whether they can connect a failed notification to its delivery attempt, responsible service, and attributable work. For a developer tools notification service, the deciding constraint is whether an incident responder can explain which delivery failed and where the work accumulated, even when a health endpoint reports success.

TL;DR: Require three evidence checks: a task outcome, an attempt history, and a cost attribution record. Use health checks to answer their declared availability question; require delivery evidence before interpreting them as proof that notifications reached their destination. A dashboard full of green tiles is a poor substitute for that distinction.

This is a selection runbook for an operational investigation, with cost attribution as the decision axis. It does not rank products. The useful comparison is between the questions each candidate can answer using the same controlled failure, because a feature list cannot tell the responder which page should have fired.

How should backend task monitoring tools complement health check APIs?

What page fired?

Start the investigation there, then write down exactly what the page asserted. Did a worker stop answering, did a task miss its deadline, or did an intended notification fail to reach a terminal delivery outcome? Treat these as separate questions in the evaluation. Otherwise, a tool can pass a health check demonstration while never being asked to account for the job the service exists to perform.

Consider a proposed test fixture: one logical notification, two delivery attempts, and one final outcome. These are test inputs, not production measurements. Give both attempts the same logical notification identifier, give each attempt its own identifier, and attach the team responsible for the notification at task creation. Now arrange for the first attempt to fail and the second to succeed. The desired accounting result is one delivered notification with two attempts of work, rather than two notifications or one unexplained failure.

Count the attempts. Don't count them as separate deliveries.

That small distinction matters to cost attribution: assigning all work only to final failures would omit the failed first attempt in this fixture, while assigning a new notification to every attempt would inflate the logical delivery count. Require candidates to preserve both views. A health result should remain a separate observation whose meaning is stated explicitly, such as whether a particular process answered a probe.

Before examining any chart, ask the evaluator to reconstruct this fixture from the stored evidence. If reconstruction depends on remembering which worker happened to execute the attempt, the evidence model needs attention before the dashboard does.

Define the three evidence checks before choosing tools

Use an investigation contract that separates business outcome, execution history, and attribution. The following table describes proposed acceptance requirements; it makes no claim about a particular product's capabilities.

Evidence check Required answer for the test fixture Selection consequence
Task outcome One logical notification eventually delivered Reject an outcome view that counts attempts as notifications
Attempt history First attempt failed; second attempt succeeded Require a way to inspect both attempts under the same task
Cost attribution Both attempts belong to the team assigned at creation Require exportable work records with a stable attribution key

OpenTelemetry's logs concepts describe log records and their relationship to other telemetry, including correlation through trace and span identifiers. That is a useful foundation for connecting execution evidence. It does not supply the notification service's definition of delivery, its team ownership rules, or its accounting policy. Write those definitions yourself and make the candidate demonstrate that it can retain them.

For this runbook, define delivery as the terminal outcome your service contract actually promises. If that contract ends at acceptance by another system, label the outcome as acceptance; do not rename it recipient delivery just to simplify the chart. This boundary determines what the page can honestly claim and what a postmortem can establish. The attribution key deserves equal scrutiny. Require the team assignment recorded when the task was created, and retain subsequent ownership changes separately if they matter to your workflow. Looking up today's team during an investigation would answer a different question from who owned the work when it entered the system.

Implement the evidence path without making it the delivery gate

Introduce the records as an additive instrumentation change, with an explicit policy for what happens when evidence cannot be exported. For the proposed rollout, keep notification delivery independent of telemetry export and make export failures observable through a separate signal. This chooses delivery continuity over guaranteed immediate accounting completeness. State that trade-off in the acceptance criteria, including who investigates missing records.

Do not silently describe incomplete records as a complete bill.

Use a stable logical task identifier, a distinct attempt identifier, the attribution key, and a clearly named outcome. Include timestamps needed for the investigation, but avoid placing notification bodies or recipient addresses into the evidence merely because they are available. The responder needs a joinable execution history; copying message content creates another data handling obligation without answering which attempt failed.

Keep the monetary calculation separate from the execution record. Record the units your accounting policy needs and identify their meaning: an attempt count can support an allocation rule, but it is not itself a measured provider charge. Where you cannot tie a charge or resource measurement to a task, label the allocation as an estimate and document its basis. An exact-looking total built from an unstated assumption is especially unhelpful during a postmortem.

There is a practical selection trade-off here. Detailed records give the responder more reconstruction options, while collecting and retaining them adds operational work. Evaluate an explicit retention window against your investigation needs, and ask what evidence remains after that window. If a candidate retains only aggregate counts for the period in question, test whether those counts still answer the attribution question before accepting that compromise.

Keep opaque task identifiers in detailed evidence rather than requiring every identifier to become a separate dashboard series. Ask for team-level summaries that lead back to individual records. That is an interface requirement worth testing with the people who will carry the page, because an aggregate without an investigation path leaves them guessing.

Verify the failure path and the attribution result

Run the two-attempt fixture through each candidate with the same definitions. First inspect the terminal task outcome. Then reconstruct the attempt sequence and check that both work records retain the original team assignment. Finally, compare the health response with the delivery evidence and explain what each observation proves. A passing health response should not erase the recorded failed attempt.

Test missing evidence too. Deliberately interrupt telemetry export in the evaluation environment and verify that the chosen delivery policy holds, that the missing-evidence signal becomes visible, and that accounting presents the resulting gap honestly. Specify the expected behavior before the test; do not let an attractive chart redefine success after the fact.

For alert acceptance, require an operator to name the condition that should page and the evidence that would resolve it. A failed attempt that later succeeds and a task that never reaches its required outcome deserve different treatment under the service's own policy. Define that policy before tuning thresholds. No universal numerical threshold follows from this fixture.

Use the resulting reconstruction as the evaluation artifact. It should contain the task's promised outcome, its two attempt records, the attribution rule, and any evidence gap.

A screenshot isn't enough.

The selection decision needs a repeatable explanation of a delivery failure and its associated work.

Roll back instrumentation while preserving the investigation

Make the added emission path independently disableable during rollout, and preserve existing delivery behavior when disabling it. If the change interferes with notification processing, disable the new path, retain evidence already collected, and mark the affected accounting interval as incomplete. Avoid changing the delivery retry policy at the same time; otherwise, an attribution comparison would mix a new observation method with a different workload.

Keep the gap visible.

After rollback, repeat the fixture against the surviving evidence path and document which of the three checks can still be satisfied. The operational recommendation remains straightforward: choose monitoring by its ability to explain the failed notification, preserve the attempt history, and attribute the work under a stated policy. A healthy endpoint is useful evidence within its declared scope. It is insufficient grounds for closing a delivery incident.

References

Top comments (0)