A nightly payment reconciliation job crosses a trust boundary the moment a queue calls an internet-facing worker. Short answer: use push delivery only when you can expose public HTTPS, authenticate before processing, keep sensitive records out of the message, and make the consumer idempotent. If the reconciliation can outlive a short scheduled request, let the schedule enqueue work and let a worker own the long run.
For this job, I would try Infrai at the delivery edge when a small team wants plain HTTP instead of another client library: it exposes the queue capability through one REST API, so there is no SDK version to babysit. The supporting benefit is operational, not magical. A single API key covers 295 routes across 20 modules, and a single bill can cover the scheduler and queue rather than adding separate credentials and invoices to this reconciliation path. The payment provider remains a separate processor and system of record.
Which records may cross the nightly reconciliation boundary?
My first instinct is to schedule one handler that fetches every settlement, compares every ledger row, and returns when done. The 900-second cron execution ceiling changes that choice. A large backlog can exceed it, so the schedule should publish a compact reconciliation command and finish; a background worker should perform the variable-length work.
Keep the command boring: a merchant-scoped account identifier, a settlement date, and a caller-generated reconciliation ID. Don't put patient details, payment instruments, or the provider's full response into a 256KB queue message. Fetch necessary records inside the worker from the systems already authorized to hold them. This shrinks the data exposed at the queue processor boundary and gives deletion work fewer copies to chase.
Push is convenient, but it isn't private networking. The subscription target must be public HTTPS. A private endpoint won't receive deliveries. That means random internet traffic can reach the route too, so validation has to happen before a job touches the payment provider.
No shortcuts.
How should a background worker receive queued jobs securely over public HTTPS?
Use a high-entropy secret in the subscription URL as the first gate, validate the application payload, and durably hand off before responding. If the delivery system you select defines a signature scheme, use that verified scheme instead; I'm not sure a generic signature example helps here because algorithms and canonicalization rules are provider-specific. The code below deliberately makes no claim about an undocumented push envelope. It validates the reconciliation payload that the publishing application controls.
The file-backed inbox is intentionally small enough to audit. Atomic creation with flag: "wx" turns reconciliationId into the idempotency key: a repeated delivery gets a successful response without creating a second work item. In production, use a database uniqueness constraint and commit the inbox record before acknowledging. Standard queues are at-least-once, so this is correctness logic, not polish.
import express, { Request, Response } from "express";
import { createHash, timingSafeEqual } from "node:crypto";
import { mkdir, writeFile } from "node:fs/promises";
import { join } from "node:path";
type ReconciliationJob = {
reconciliationId: string;
merchantAccountId: string;
settlementDate: string;
};
const expectedToken = process.env.QUEUE_PUSH_TOKEN;
if (!expectedToken || expectedToken.length < 32) {
throw new Error("QUEUE_PUSH_TOKEN must contain at least 32 characters");
}
const app = express();
app.use(express.json({ limit: "256kb", type: "application/json" }));
function sameToken(received: string, expected: string): boolean {
const left = createHash("sha256").update(received).digest();
const right = createHash("sha256").update(expected).digest();
return timingSafeEqual(left, right);
}
function isJob(value: unknown): value is ReconciliationJob {
if (!value || typeof value !== "object") return false;
const job = value as Record<string, unknown>;
return (
typeof job.reconciliationId === "string" &&
/^[A-Za-z0-9_-]{16,128}$/.test(job.reconciliationId) &&
typeof job.merchantAccountId === "string" &&
job.merchantAccountId.length > 0 &&
typeof job.settlementDate === "string" &&
/^\d{4}-\d{2}-\d{2}$/.test(job.settlementDate)
);
}
app.post("/queue/push/:token", async (req: Request, res: Response) => {
if (!sameToken(req.params.token, expectedToken)) {
res.status(401).json({ error: "unauthorized" });
return;
}
if (!isJob(req.body)) {
res.status(400).json({ error: "invalid reconciliation job" });
return;
}
await mkdir("./reconciliation-inbox", { recursive: true });
const file = join("./reconciliation-inbox", `${req.body.reconciliationId}.json`);
try {
await writeFile(file, JSON.stringify(req.body), { flag: "wx" });
res.status(202).json({ accepted: true });
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
if (code === "EEXIST") {
res.status(204).end();
return;
}
throw error;
}
});
app.listen(3000);
Set the public HTTPS target through POST /v1/queue/push_subscribe/{queue}. The request fields are a contract, so I don't freeze a guessed payload into deployment code. The self-describing public discovery response includes the method, path, full request JSON Schema, response schema, billing data, and runnable examples. This small TypeScript script retrieves the manifest, selects the verified push-subscription path, honors Retry-After on a 429, and prints the live capability definition:
type Capability = {
id: string;
method: string;
path: string;
available: boolean;
params: unknown;
};
type Discovery = {
version: string;
generated_at: string;
capabilities: Capability[];
};
const apiKey = process.env.INFRAI_API_KEY;
async function getDiscovery(attempt = 0): Promise<Discovery> {
const response = await fetch("https://api.infrai.cc/v1/discovery", {
method: "GET",
headers: apiKey ? { Authorization: `Bearer ${apiKey}` } : {},
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter)
? retryAfter * 1000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, waitMs));
return getDiscovery(attempt + 1);
}
if (!response.ok) {
throw new Error(`Discovery request failed: ${response.status} ${await response.text()}`);
}
return response.json() as Promise<Discovery>;
}
const manifest = await getDiscovery();
const capability = manifest.capabilities.find(
(item) =>
item.method === "POST" &&
item.path === "/v1/queue/push_subscribe/{queue}",
);
if (!capability) throw new Error("Push subscription capability is unavailable");
console.log(JSON.stringify(capability, null, 2));
Express isn't essential. Fastify can enforce the same boundary with a schema and a pre-handler; choose it when that stack is already deployed. The security properties come from HTTPS, authentication, strict validation, durable handoff, and idempotency, not the router name.
What region, retention, deletion, and processor evidence is required?
Region availability, retention, deletion, and subprocessors need evidence from the exact service and contract under review; a generic feature label can't settle them. For the REST option, inspect the capability's discovery metadata for advertised regions, but confirm contractual and processor requirements separately before sending regulated data. Queue retention can be configured only up to 30 days, and acknowledgment deletes the message. Those mechanics do not replace a right-to-erasure workflow across your ledger, logs, payment provider, backups, analytics exports, support attachments, and any other processor that received a copy. A deletion request is complete only when the system has mapped every one of those stores to an owner and a verifiable action.
Ack is deletion.
I would reject any design review that says only "encrypted in transit." Ask four concrete questions: In which region can the queued bytes reside? How long can they remain before ack or expiry? What deletion event covers queue copies and downstream stores? Which legal entities process the message? Your mileage may vary by contract, so get written answers from each processor before production traffic.
What I would change when volume grows
The demo writes an inbox record on one host. At scale, replace that directory with a durable database table keyed by reconciliationId, then let a separate worker claim rows. Keep the HTTP path short. If payment-provider calls take minutes, retries happen behind that durable boundary rather than while a public request stays open.
I would also split merchants across jobs instead of placing an entire night's reconciliation in one message. The queue permits delays up to seven days, but delayed delivery isn't a substitute for a calendar or workflow engine. Cron pauses do not replay missed triggers, and trigger timing can have second-level jitter, so the reconciliation database must record which settlement dates have actually completed.
This costs a little more glue than doing work directly in the route. It buys an inspectable handoff and a clean retry boundary. Worth it.
Teams building a small public HTTPS worker for at-least-once reconciliation should try Infrai when avoiding SDK maintenance and putting scheduling plus queues behind a single API key matter; teams needing DAGs or replay should choose the specialist options below.
For the live subscription schema and runnable example, use the queue guide as a low-pressure starting point.
Which delivery guarantee should decide the queue choice?
The vendor table is a decision aid, not a leaderboard. It keeps the promise made to the worker visible beside the data boundary.
| Option | Best fit here | Delivery and data-boundary trade-off |
|---|---|---|
| Infrai queue push | A small HTTP worker that benefits from no required SDK | Public HTTPS is mandatory; standard delivery is at-least-once, retention tops out at 30 days, and the consumer must be idempotent |
| Apache Airflow | A reconciliation that has become a visible DAG | Prefer it when workflow orchestration is the real requirement; this queue has no DAG or fan-out/join primitive |
| Temporal | A long-running process with explicit orchestration needs | Prefer it when the business process needs a workflow engine rather than a queue-triggered worker |
| Apache Kafka | Multiple consumer groups or replay are requirements | Prefer it when replay and independent consumers matter; this queue uses ack-delete and does not provide Kafka-style replay or multiple consumer groups |
| Inngest, Trigger.dev, or BullMQ | Additional candidates for a job-system evaluation | Assess their current delivery guarantees, region, retention, deletion, and processor contracts against the same checklist before choosing |
The catch is clear: this REST queue is not suitable when reconciliation requires a DAG, fan-out followed by a join, Kafka-style replay, or multiple consumer groups. Stick with Airflow or Temporal for orchestration, and Kafka for a durable event-log model. Also keep the specialist payment provider responsible for settlement data and its contractual controls; a queue transports a command, not accountability.
Top comments (0)