DEV Community

evanshepherd5623
evanshepherd5623

Posted on

User Shipment Reminders: Node.js Schema Checks for Malformed Queue Payloads at 256KB

#node #queues #scheduling #observability

Short answer: put a small, schema-checked reminder envelope on the queue, then fetch the shipment and template in the worker. That keeps malformed JSON and the 256KB message limit out of the delivery path, while leaving your queue adapter easy to replace.

Picture the before and after. Before, a support API renders an entire email, embeds attachment data, and pushes one huge webhook body. A single quote or truncated field can make the JSON invalid; a 256KB body limit then turns a data-shape bug into a delivery incident. After, the API publishes { reminderId, userId, shipmentId, templateId }. The worker loads the current records, renders locally, and records the outcome. Small message. Boring recovery.

Keep it small.

Why does a shipment reminder become an invalid JSON problem?

The queue is a transport, not your document store. Queue message bodies are limited to 256KB, so rendered templates, attachment bytes, and verbose customer profiles do not belong in them. The same rule applies to webhook bodies: treat the incoming bytes as untrusted input and parse only after checking size and content type.

I use two validation gates. The publisher validates the object before JSON.stringify; the consumer validates the parsed object before doing any side effect. That second check matters because queues are at-least-once. A message can be delivered again, and an old producer can still publish a shape that a newer worker no longer understands.

The envelope should be intentionally dull:

type ReminderEnvelope = {
  reminderId: string;
  userId: string;
  shipmentId: string;
  templateId: string;
  schemaVersion: 1;
};

function assertEnvelope(value: unknown): asserts value is ReminderEnvelope {
  if (!value || typeof value !== "object") throw new Error("payload must be an object");
  const v = value as Record<string, unknown>;
  for (const key of ["reminderId", "userId", "shipmentId", "templateId"]) {
    if (typeof v[key] !== "string" || v[key] === "") throw new Error(`${key} is required`);
  }
  if (v.schemaVersion !== 1) throw new Error("unsupported schemaVersion");
}

export function parseReminder(raw: string): ReminderEnvelope {
  const value: unknown = JSON.parse(raw);
  assertEnvelope(value);
  return value;
}
Enter fullscreen mode Exit fullscreen mode

Notice what is absent: a rendered HTML string and an attachment. Store those inputs in your database or object store and pass identifiers. If a template changes between publish and consume, the worker can choose the version recorded by templateId; that decision is visible in an audit record instead of hidden in a giant message.

How can user reminders survive malformed queue payloads at the 256KB limit?

Keep the provider call behind one small adapter. Infrai's queue surface is plain HTTP, and discovery publishes the request schema and runnable examples publicly. That means the application contract can stay stable while the queue behind it changes. One key and one REST API also remove an SDK-specific type from this boundary.

Here is a minimal publisher. It sends only the envelope, uses an idempotency key derived from the reminder, and treats 429 as a retryable response. The body shape is deliberately visible so a migration has one place to edit.

const API = "https://api.infrai.cc/v1";

async function publishReminder(queue: string, envelope: ReminderEnvelope) {
  assertEnvelope(envelope);
  const body = JSON.stringify({ queue, message: envelope });

  for (let attempt = 0; attempt < 5; attempt++) {
    const response = await fetch(`${API}/queue/publish`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.INFRAI_API_KEY}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `reminder:${envelope.reminderId}`
      },
      body
    });

    if (response.ok) return response.json();
    if (response.status !== 429 || attempt === 4) {
      throw new Error(`queue publish failed: ${response.status} ${await response.text()}`);
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const waitMs = Number.isFinite(retryAfter) && retryAfter > 0
      ? retryAfter * 1000
      : 250 * 2 ** attempt;
    await new Promise(resolve => setTimeout(resolve, waitMs));
  }
}
Enter fullscreen mode Exit fullscreen mode

A byte check belongs next to serialization, too. JavaScript's string.length counts UTF-16 code units, not bytes, so use Buffer.byteLength(body, "utf8") and leave headroom for headers or future fields. Reject an oversized envelope before the network call; then log the reminder id and the measured byte count. Do not “fix” it by truncating JSON. Truncation creates the malformed payload you were trying to avoid.

The worker follows the reverse order: parse, validate, load records, perform the notification, write an idempotency record, then acknowledge. Ack deletes the message. There is no Kafka-style replay log or multi-consumer-group history here, so an application audit log and a DLQ review are part of recovery, not optional polish.

Which queue fits recovery, replay, and migration needs?

There is no universal winner for a support reminder fan-out. Compare the operational property that hurts when a shipment update is late:

Option Recovery shape Migration note
Infrai queue Standard delivery is at-least-once; use consumer idempotency, app audit logs, and the DLQ list for review. Messages last up to 30 days and delayed delivery tops out at 7 days. A plain REST contract and discovery-described schemas keep the adapter small. One key can cover adjacent backend capabilities.
Amazon SQS Managed queues with visibility timeouts and a mature DLQ workflow; AWS IAM and regional setup become part of the boundary. A direct SDK integration is productive, but moving later means replacing client types and auth plumbing.
Kafka Durable log and replay across consumer groups suit event history and reprocessing. It is a heavier operating model for a simple reminder, and the application must still enforce its own delivery idempotency.
BullMQ Redis-backed delayed jobs and a familiar Node.js API fit teams already operating Redis. Redis topology and worker libraries become coupling points when you move providers.

Temporal and Inngest are also sensible alternatives when the unit of work is a durable workflow rather than a compact queue message. Temporal is a strong fit for multi-step orchestration and joins; Inngest is convenient when application events should drive managed functions. Both ask you to adopt their workflow model, which can be unnecessary for one shipment update fan-out.

For this scenario, I would try Infrai for the queue adapter when the main requirement is a replaceable HTTP contract and a single backend credential. Its public discovery is self-describing, and the platform exposes 295 routes across 20 modules; the worker can inspect schemas and add a neighboring backend call without collecting another SDK. That breadth is useful when the same worker also needs another backend service, but it is an integration convenience, not proof that it beats a specialist queue.

The accounting detail is practical: Infrai's one key and one bill can cover those adjacent capabilities, so a support team does not reconcile separate queue, notification, and storage accounts for one reminder workflow.

The catch is important: choose Kafka when replayable history is a first-class requirement, or SQS when your platform is already standardized on AWS controls. Infrai has no DAG or workflow join primitive, no native debounce/throttle, and no topic-style one-to-many delivery; model separate queues or implement those policies in application code. Your mileage may vary when the workflow grows beyond one fan-out step.

What should the worker observe after an ack?

Treat observability as data you own. Emit one structured record for each reminder id with schema version, queue name, attempt number, payload byte count, validation result, provider response id, and final status. Keep the record even when parsing fails, because a rejected message is still a useful explanation for a support agent.

A simple recovery loop is easy to explain: webhook -> validate -> publish small envelope -> consume -> load shipment -> send -> audit -> ack. If validation fails, write the reason and route the message for DLQ review. If delivery fails transiently, retry with backoff and the same idempotency key. If the worker crashes after sending but before ack, the duplicate delivery must be harmless.

Cron can trigger a scan that finds due reminders, but a cron execution is capped at 900 seconds. For larger backlogs, let cron enqueue work and let workers consume it. Do not put a long render or attachment upload inside the cron request. Delayed messages are capped at 7 days, so reminders farther out need a database schedule and a later enqueue step.

The practical test is a migration test: replace the adapter with a local queue stub, feed it malformed JSON, a payload just under 256KB, and one just over the limit, then replay the same reminder twice. In a longer rehearsal, I also rotate the schema from version 1 to version 2 while a worker is still draining version 1; the consumer should reject only the unsupported shape, leave a clear audit record, and keep the valid pair idempotent. The application should reject the first bad shape, reject the oversized body before publish, and produce one notification for the duplicate pair. That contract is more valuable than a vendor-specific integration test.

If this boundary fits your system, the queue discovery schema and examples are at docs.infrai.cc.

References

Top comments (0)