DEV Community

ViggoKnight2318
ViggoKnight2318

Posted on

Shipment Fan-Out — Checking Delayed Webhook Payload Formatting Issues

#node #webhooks #observability #distributedsystems

TL;DR: validate and normalize a shipment event before it enters a delayed queue, then log the exact schema version and delivery attempt at the fan-out boundary. In a fintech system, that small ordering choice protects latency-sensitive subscribers from malformed payloads without making every consumer rediscover the same rules.

The useful mental model is a two-stage pipeline. Before: accept an arbitrary webhook-shaped object, wait 30 seconds, and discover at delivery time that trackingNumber is nested under a different key. After: turn the inbound event into one canonical envelope, delay that envelope, and send a separately validated delivery payload to each subscriber. The queue schedules work. It should not be the place where meaning is invented.

Where does the formatting failure actually happen?

A delayed webhook has at least three representations: the request received from a carrier, the message stored for later work, and the HTTP body sent to a subscriber. Treating those as the same object is the trap. A carrier may send a timestamp as a string, the queue serializer may preserve it as JSON text, and a subscriber may require an RFC 3339 timestamp plus a stable event name. None of those transformations is mysterious. They are just easy to leave implicit.

For a shipment-update fan-out, I keep the queue message deliberately boring. It contains an event ID, an immutable shipment reference, the status, the event time, and a schema version. Subscriber-specific headers, signatures, and retry metadata belong at delivery time. This separation matters because a delayed job can be replayed, while a signature is normally tied to the exact body and delivery moment.

Here is a compact TypeScript boundary. It rejects missing values and refuses to silently coerce an invalid date. That is a better failure than publishing a payload that looks plausible.

type ShipmentStatus = "in_transit" | "delivered" | "exception";

type ShipmentEvent = {
  eventId: string;
  shipmentId: string;
  status: ShipmentStatus;
  occurredAt: string;
  schemaVersion: 1;
};

function toShipmentEvent(input: unknown): ShipmentEvent {
  if (!input || typeof input !== "object") {
    throw new Error("shipment event must be an object");
  }

  const value = input as Record<string, unknown>;
  const status = value.status;
  const occurredAt = value.occurredAt;

  if (typeof value.eventId !== "string" || value.eventId.length === 0) {
    throw new Error("eventId is required");
  }
  if (typeof value.shipmentId !== "string" || value.shipmentId.length === 0) {
    throw new Error("shipmentId is required");
  }
  if (status !== "in_transit" && status !== "delivered" && status !== "exception") {
    throw new Error("unsupported shipment status");
  }
  if (typeof occurredAt !== "string" || Number.isNaN(Date.parse(occurredAt))) {
    throw new Error("occurredAt must be a parseable timestamp");
  }

  return {
    eventId: value.eventId,
    shipmentId: value.shipmentId,
    status,
    occurredAt: new Date(occurredAt).toISOString(),
    schemaVersion: 1,
  };
}
Enter fullscreen mode Exit fullscreen mode

The important detail is not the helper itself. It is where the helper runs: before enqueueing. A malformed event can still be recorded for investigation, but it does not become a delayed delivery job by accident.

That boundary is the whole trick.

How can you check delayed webhook payload formatting issues before fan-out?

Use a stable envelope rather than passing the original request through unchanged. The queue payload can look like this:

type DelayedShipmentJob = {
  kind: "shipment.updated";
  event: ShipmentEvent;
  availableAt: string;
};

const job: DelayedShipmentJob = {
  kind: "shipment.updated",
  event: toShipmentEvent(incomingBody),
  availableAt: new Date(Date.now() + 30_000).toISOString(),
};
Enter fullscreen mode Exit fullscreen mode

A consumer then has a narrow contract: read kind, inspect event.schemaVersion, and deliver the event to the subscribers selected for that shipment. It should not guess whether status came from state, whether occurredAt is local time, or whether the body was already stringified. One JSON encoding is enough. Double encoding is a classic symptom: the receiver parses the body once and gets a string containing JSON instead of an object.

This is the formatting check I want visible in a trace: raw input enters the normalizer, the canonical object is logged by ID and version, serialization happens once, and the delivery renderer produces the final bytes. If the receiver reports a shape error, the trace points to one boundary instead of asking an on-call engineer to compare three opaque blobs. The extra fields are small, but the debugging path is much shorter.

I also record the payload shape without recording sensitive shipment details. A useful log line includes eventId, schemaVersion, subscriberId, attempt, queueDelayMs, and a validation result. It should exclude authentication material and avoid dumping the full body. Logs answer “which shape failed?” without becoming a second copy of the customer data.

For metrics, separate queue delay from delivery duration. Queue delay is the time between occurredAt or enqueue time and the worker starting. Delivery duration is the time spent calling one subscriber. If those values share one histogram, a slow endpoint can look like a scheduling problem and lead to the wrong fix.

Can retries preserve correctness without slowing every subscriber?

Yes, if the job is idempotent and the retry decision is explicit. A timeout is not proof that the subscriber did not receive the event. Include the stable eventId in the request and let the receiver deduplicate it. Keep the same event body across retries; changing the timestamp or generating a new ID turns one delivery into several logical events.

A practical delivery record tracks the event ID, subscriber ID, attempt number, response class, and next eligible time. A 2xx response completes that subscriber delivery. A temporary transport failure or a 5xx response can be retried with bounded backoff. A 4xx response usually needs classification: malformed authentication and unsupported schema are not helped by repeating the same request forever.

The fan-out decision is where latency and cost meet. Sending to every subscriber in parallel reduces wall-clock latency, but it increases concurrent connections and the chance that a burst exhausts a worker pool. A bounded concurrency limit gives you a visible control knob. For example, process 50 subscriber deliveries per worker, measure the resulting queue delay, and adjust from observed latency objectives rather than from a convenient round number. The number is a starting configuration, not a universal target.

The event should be accepted once, then each subscriber should have an independent delivery state. If one bank's risk system is slow, the notification for another subscriber should not be held hostage by a shared “all delivered” flag. That is the difference between fan-out as a workflow and fan-out as a single fragile HTTP call.

Keep the failure local.

How do you test a payload that will be delivered later?

Test the boundaries in the order they run. First, feed the normalizer representative carrier input and assert the exact canonical object. Include a missing field, an unknown status, an invalid timestamp, and an already-stringified body. Second, serialize and deserialize the delayed job. Third, render the subscriber request and assert its content type, signature input, and body bytes. Finally, replay the same eventId and verify that the delivery state does not create a duplicate side effect.

A small contract fixture catches more than a large collection of happy-path snapshots. Keep one fixture for each schema version. When version 2 arrives, make the compatibility rule visible: either migrate at enqueue time or let the worker handle both versions for a defined period. Do not silently accept every unknown field and call that forward compatibility; permissive parsing can hide a producer typo.

The queue itself adds another operational constraint. Ordered delivery and deduplication semantics are useful tools, but they do not replace an application-level idempotency key. Queue documentation describes the behavior and limits of FIFO queues, including message groups and deduplication windows; those guarantees still need to be matched to the shipment event's business identity.

A decision rule for production

Choose the simplest path that meets the latency objective while keeping payload ownership clear. If a subscriber needs near-real-time updates, route it through the same canonical event but use a short delay or immediate work. If a downstream system is sensitive to bursts, keep the delayed queue and bound concurrency. If schema errors are frequent, move validation earlier and add a metric for rejected events instead of increasing retries.

The before/after check is straightforward: before, a worker learns about formatting trouble after waiting; after, ingestion rejects or quarantines the bad shape, the queue stores a versioned event, and delivery logs explain each attempt. That sequence makes a malformed webhook observable, replayable, and isolated from unrelated subscribers. It also gives the team a clean trade-off: spend concurrency to reduce latency, or spend queue time to protect downstream capacity.

References

Top comments (0)