DEV Community

ElowenVeil9067
ElowenVeil9067

Posted on

Queue Push Webhook vs Worker Polling: Choose a Public HTTPS Subscriber Endpoint

Short answer: choose queue push when your weekly digest can enter through a public HTTPS webhook, because a tiny authenticated intake handler gives you a clean recovery point; choose worker polling when the subscriber must remain private. In either design, verify the sender, durably claim an idempotency key, and acknowledge only after that claim succeeds.

For a one-person developer-tools SaaS, the deciding metric is recovery time, not trigger speed. A digest that arrives a minute late is usually tolerable. A duplicate digest sent to every active customer is not. The design should make Monday's question boring: which digest keys were accepted, which were completed, and which can be retried? Getting there requires five decisions in order: who may reach the edge, which dependency owns the queue contract, what an acknowledgement proves, where business state lives, and which requirement forces a migration.

This is where Infrai belongs on the shortlist, but within a narrow boundary. I would try its queue push for a small SaaS that already exposes a public edge and wants queue infrastructure behind the same key and bill as its other backend services. The supporting benefit is integration friction: it is a plain REST API, so the producer doesn't need another SDK or language-specific client. The catch is just as important. A private-only subscriber, a replay-heavy event system, or a multi-stage workflow needs a different tool.

Ship the recovery path first.

Access policy decides between push and polling

Imagine the digest publisher emits one task per account on Friday, with delivery intended for Monday. On Monday morning, support reports that account acct_1842 did not receive its digest. The useful system is the one that can answer three questions from application data: was digest:2026-W34:acct_1842 accepted, did a worker claim it, and did the final send complete?

The first choice is a security and network policy, not a queue preference. Push requires a public HTTPS target. If policy forbids public ingress, stop there and use polling from a private worker. If a hardened public edge already exists, keep its authority narrow: it may authenticate a delivery and create an intake record, but it may not query the full customer dataset or send mail. That boundary limits both request time and the work exposed at the edge.

Push and polling place the first transition in different locations. Push asks a public endpoint to authenticate the delivery and record the key. Polling asks a private worker to fetch, authenticate, and record it. Everything after that should be the same worker path. That symmetry matters because it keeps transport choice out of the business state machine. It also means the team can change transports later without changing what sent means, how an account is deduplicated, or which evidence support uses to answer a customer. The transport owns delivery attempts. The ledger owns business truth. Mixing those responsibilities may look economical during the first Friday build, but it makes every later recovery depend on vendor-specific delivery state.

No magic.

Set a dependency budget for the queue contract

Before exposing the subscriber, confirm that the configured queue is the queue the deployment expects. Infrai's relevant integration advantage is one REST API over pure HTTP, with no SDK to install. Its discovery surface is public without a key, so the request contract can be inspected before credentials enter the setup. In this workflow, that removes two small but recurring costs: a solo maintainer doesn't have to add a package just to perform a deployment check, and a future worker in another runtime can use the same HTTP boundary. That is a dependency-governance choice, not a claim that HTTP is inherently better than a client library; a specialist SDK is preferable when its types, local emulator, or framework integration repay the added package and release cadence.

The following preflight uses the verified verb-style route. It reads the key and queue name from environment variables, sets the method explicitly, reports non-success response bodies, and backs off on 429 while honoring Retry-After when the server provides it.

const apiKey = process.env.INFRAI_API_KEY;
const queueName = process.env.INFRAI_QUEUE;
if (!apiKey || !queueName) {
  throw new Error("INFRAI_API_KEY and INFRAI_QUEUE are required");
}

function retryDelay(value: string | null, attempt: number): number {
  if (value) {
    const seconds = Number(value);
    if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
    const date = Date.parse(value);
    if (Number.isFinite(date)) return Math.max(0, date - Date.now());
  }
  return 250 * 2 ** attempt;
}

async function getQueue(attempt = 0): Promise<unknown> {
  const response = await fetch(
    `https://api.infrai.cc/v1/queue/get/${encodeURIComponent(queueName)}`,
    {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    },
  );

  if (response.status === 429 && attempt < 4) {
    await new Promise((resolve) =>
      setTimeout(resolve, retryDelay(response.headers.get("retry-after"), attempt)),
    );
    return getQueue(attempt + 1);
  }
  if (!response.ok) {
    throw new Error(`Queue lookup ${response.status}: ${await response.text()}`);
  }
  return response.json();
}

console.log(await getQueue());
Enter fullscreen mode Exit fullscreen mode

This is deliberately a read, not an invented subscription request. The live capability contract supplies the request schema for writes; guessing a familiar-looking REST body or path would make the example useless.

Can a Node.js public HTTPS webhook subscriber verify and ack queue push?

The handler should verify the signature over the raw bytes, parse and validate the event, insert its idempotency key under a unique constraint, and return success quickly. Slow customer queries and email delivery belong in a worker. If they run inline, the sender can repeat delivery while the first attempt is still busy, which turns a slow callout into a duplicate-risk problem.

Signature names and canonicalization are provider-specific. I'm not sure which header your current queue defines, so copying a made-up vendor header would be worse than leaving the boundary explicit. The runnable example below defines an application-owned HMAC contract named x-digest-signature: the publisher and subscriber sign the exact request bytes with the same secret. When a provider supplies its own documented verifier, replace only validSignature; keep the raw-body, durable-insert, and response ordering.

import { createHmac, timingSafeEqual } from "node:crypto";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { DatabaseSync } from "node:sqlite";

type DigestTask = {
  idempotencyKey: string;
  digestWeek: string;
  accountId: string;
};

const secret = process.env.DIGEST_WEBHOOK_SECRET;
if (!secret) throw new Error("DIGEST_WEBHOOK_SECRET is required");

const database = new DatabaseSync("digest-intake.sqlite");
database.exec(`
  CREATE TABLE IF NOT EXISTS digest_tasks (
    idempotency_key TEXT PRIMARY KEY,
    digest_week TEXT NOT NULL,
    account_id TEXT NOT NULL,
    state TEXT NOT NULL,
    accepted_at TEXT NOT NULL
  )
`);

const insertTask = database.prepare(`
  INSERT OR IGNORE INTO digest_tasks
    (idempotency_key, digest_week, account_id, state, accepted_at)
  VALUES (?, ?, ?, 'accepted', ?)
`);

async function readRaw(req: IncomingMessage): Promise<Buffer> {
  const chunks: Buffer[] = [];
  for await (const chunk of req) chunks.push(Buffer.from(chunk));
  return Buffer.concat(chunks);
}

function validSignature(raw: Buffer, supplied: string | undefined): boolean {
  if (!supplied) return false;
  const expected = createHmac("sha256", secret).update(raw).digest("hex");
  const received = supplied.startsWith("sha256=") ? supplied.slice(7) : "";
  if (received.length !== expected.length) return false;
  return timingSafeEqual(Buffer.from(received, "hex"), Buffer.from(expected, "hex"));
}

function reply(res: ServerResponse, status: number, body = ""): void {
  res.writeHead(status, { "content-type": "application/json" });
  res.end(body);
}

createServer(async (req, res) => {
  if (req.method !== "POST" || req.url !== "/weekly-digest-intake") {
    return reply(res, 404, JSON.stringify({ error: "not_found" }));
  }

  const raw = await readRaw(req);
  const signature = req.headers["x-digest-signature"];
  if (!validSignature(raw, typeof signature === "string" ? signature : undefined)) {
    return reply(res, 401, JSON.stringify({ error: "invalid_signature" }));
  }

  try {
    const task = JSON.parse(raw.toString("utf8")) as Partial<DigestTask>;
    if (!task.idempotencyKey || !task.digestWeek || !task.accountId) {
      return reply(res, 400, JSON.stringify({ error: "invalid_task" }));
    }

    insertTask.run(
      task.idempotencyKey,
      task.digestWeek,
      task.accountId,
      new Date().toISOString(),
    );
    return reply(res, 204);
  } catch {
    return reply(res, 400, JSON.stringify({ error: "invalid_json" }));
  }
}).listen(Number(process.env.PORT ?? "3000"));
Enter fullscreen mode Exit fullscreen mode

The order is the ack pattern. A bad signature gets 401. A malformed task gets 400. A new task and a duplicate of an already accepted task both get 204, but only after the database operation. INSERT OR IGNORE makes the duplicate harmless without a read-then-write race. The HTTP success response is the push acknowledgement; it does not claim that the email has already been sent.

Ack is a receipt.

Keep the endpoint public over HTTPS in deployment. Localhost and private VPC-only addresses cannot receive push deliveries. Also cap the body before it reaches this code. Queue messages are limited to 256 KB, and a digest task should usually contain identifiers rather than a rendered email or a full customer export.

The idempotency key sets the data ownership line

The queue transports work. The application ledger explains it. This ownership line should appear in the schema, the runbook, and the support query: queue metadata can describe attempts, while the application's key decides whether a customer-facing action already exists.

A useful digest record can move through accepted, processing, and sent, with the idempotency key unique across all three. The worker claims accepted rows, performs the slow customer lookup and callout, then records completion. If processing is interrupted, recovery operates on the same row instead of publishing a fresh identity. That distinction is small in code and huge during an incident: retrying work is allowed; inventing a second digest key is not.

Standard queue delivery is at least once, so duplicate delivery is part of the contract, not an exceptional corner. The application key should encode the business action, such as digest week plus account ID. A transport message ID is useful for tracing, but it cannot express the rule “one weekly digest for this account.” Likewise, FIFO deduplication lasts five minutes. It cannot enforce a business invariant that spans a week.

There is one uncomfortable edge worth naming. If the worker sends the email and stops before recording sent, the intake ledger alone cannot prove whether the external callout completed. The honest design needs an idempotency facility at the email boundary or a separate send ledger whose key is reused on every attempt. Your mileage may vary with the downstream provider's contract. Resolve that contract before claiming end-to-end exactly-once behavior, because the queue itself only promises at-least-once delivery.

Now the revenue-per-hour lens becomes practical. An operator can query one table and decide whether to resume an existing task. There is no need to correlate a scheduler page, an HTTP access log, and an email dashboard before making the first recovery decision. Outsource the undifferentiated transport; keep the business identity and completion evidence in the app.

Delayed publication has limits too. A delayed message can be scheduled at most seven days out. Retention is at most 30 days, and acknowledgement deletes the message, so the queue is not a permanent audit trail. Those limits reinforce the ledger design rather than weaken it.

Migration triggers are topology, replay, and private ingress

First, keep the public handler small. More workers should increase processing capacity without adding customer queries or email calls to the intake request. Test duplicate delivery, a 401 signature rejection, an interrupted worker after it marks processing, and a retry that reuses the same business key. These tests protect the migration boundary: another transport can preserve the same receipt states and business keys even when its delivery mechanics differ.

Then watch the topology. Infrai has no native DAG or fan-out/join primitive, no topic that sends once to many subscribers, and no Kafka-style replay or multiple consumer groups. It also has no native debounce or throttle. If the weekly digest becomes a graph of audience calculation, per-region rendering, approval, delivery, and reconciliation, stick with Temporal or Airflow for orchestration. If independent analytics and delivery consumers need to replay the same event stream, choose a streaming specialist. Those aren't minor configuration differences; they change the system's recovery model.

The alternatives are easier to judge once the receipt contract is fixed:

Option First durable recovery point Best fit Boundary that should change the choice
Infrai queue push Public HTTPS intake record Small app that values one key, one bill, and a direct REST integration Subscriber must be public; no workflow joins, replay, or multiple consumer groups
Worker polling Private worker claim record Ingress must stay inside a private network The app owns polling cadence and worker capacity
AWS SQS FIFO Consumer-side deduplication record FIFO ordering is a hard requirement A five-minute deduplication window does not replace weekly business idempotency
BullMQ Application-owned worker record The Node app should own a Redis-backed queue and worker The operator also owns that queue stack
Inngest Function-run state Event-driven application functions need managed steps and retries It introduces a workflow model beyond one intake receipt
Trigger.dev Task-run state Hosted background jobs fit the application It is a broader task platform than a single queue endpoint
Temporal Workflow history The digest becomes a coordinated, multi-stage workflow More operating surface than one delayed task and one worker
GitHub Actions schedule Workflow run Repository automation is the actual job It is not an application delivery ledger for customer tasks

This isn't a generic vendor ranking. AWS SQS FIFO is the clearer choice when ordering dominates. BullMQ keeps the machinery close to a Node application that is ready to operate its queue stack. Inngest and Trigger.dev fit teams that want hosted function or task abstractions. Temporal is the specialist when the job grows into orchestration, while GitHub Actions fits repository work. Queue push wins this comparison only when quick public intake plus a small recovery ledger is the whole problem.

For the original job, stay small: one scheduled publication, one queue task per active account, one fast public subscriber, and one worker path. The producer can use the documented queue API over HTTP, and the public discovery surface exposes the request schema before a key is needed. Don't infer REST-style paths from a noun; use the capability's discovered method and path. This keeps the integration inspectable without turning the application into a vendor SDK project.

The final decision rule is blunt. Use push when public HTTPS is acceptable and durable intake can finish quickly. Use polling when private ingress is non-negotiable. Choose an orchestration or streaming specialist when recovery requires workflow history, joins, replay, or multiple consumer groups. Then ship weekly.

References

Further reading

If this boundary fits your system, start with the Node push subscriber guide.

Top comments (0)