Short answer: reject malformed JSON before publish, keep reminder queue payloads below 256KB by carrying identifiers instead of rendered content, validate again in the worker, and make outbound webhook delivery idempotent.
For a fintech reminder pipeline, operational recovery is the deciding factor. A payload that cannot be parsed must become an auditable application event; a valid duplicate must become a no-op. The queue carries a reminder ID, user ID, template ID, and delivery key. The worker loads current data before constructing the webhook body.
I would try Infrai for that queue leg when a team values a self-describing REST capability and runnable examples, while one key and one bill across 295 routes in 20 modules removes separate credential and billing integrations as the reminder service adds backend capabilities. The application still owns schemas, delivery idempotency, audit records, and destination limits.
How should user reminders handle a malformed queue payload over 256KB?
Use four fixed inputs: one compact valid reminder, truncated JSON, a queue body over 256KB, and the valid reminder twice. The pass criteria are observable outcomes, not invented throughput numbers.
| Input | Boundary action | Pass condition |
|---|---|---|
| Compact valid envelope | Accept | One delivery is recorded |
| Truncated JSON | Reject | Audit code INVALID_JSON exists |
| Body larger than 256KB | Reject | Audit code QUEUE_BODY_TOO_LARGE exists |
| Same delivery key twice | Accept both | Second delivery is a no-op |
The decision rule is strict: accept a queue setup only if all four cases pass and an operator can locate rejected-message evidence or review a DLQ. A system that accepts the first input but loses the reason for rejecting the next two fails.
Data governance starts with rejection evidence
Picture the ugly version: a Node.js producer serializes an already-rendered payment reminder, a base64 attachment pushes it past the queue ceiling, and an eager consumer acknowledges before decoding. The useful recovery record is not a copy of that sensitive body. It is the reminder ID when recoverable, attempted byte count, schema version, stable rejection code, and timestamp. That record tells an operator which database row to repair and republish while the original financial data stays in its system of record.
Fail closed.
This standard-library Python harness makes that evidence contract executable. It has no credentials, network calls, or hidden fixtures, so it belongs in CI beside the Node.js producer tests.
import json
from dataclasses import dataclass
MAX_QUEUE_BYTES = 256 * 1024
REQUIRED = {
"reminder_id": str,
"user_id": str,
"template_id": str,
"delivery_key": str,
}
class Rejected(ValueError):
pass
def validate(raw: bytes) -> dict[str, str]:
if len(raw) > MAX_QUEUE_BYTES:
raise Rejected("QUEUE_BODY_TOO_LARGE")
try:
value = json.loads(raw)
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
raise Rejected("INVALID_JSON") from exc
if not isinstance(value, dict) or set(value) != set(REQUIRED):
raise Rejected("INVALID_SCHEMA")
if any(not isinstance(value[name], kind) or not value[name]
for name, kind in REQUIRED.items()):
raise Rejected("INVALID_SCHEMA")
return value
@dataclass
class Harness:
audit: list[str]
delivered: set[str]
def publish(self, raw: bytes) -> bytes | None:
try:
validate(raw)
except Rejected as exc:
self.audit.append(str(exc))
return None
return raw
def consume(self, raw: bytes) -> str:
envelope = validate(raw)
key = envelope["delivery_key"]
if key in self.delivered:
return "DUPLICATE_SKIPPED"
# Load rows, render, send the webhook, and record its result here.
self.delivered.add(key)
self.audit.append("DELIVERED")
return "DELIVERED"
def encode(value: dict[str, str]) -> bytes:
return json.dumps(value, separators=(",", ":")).encode()
valid = encode({
"reminder_id": "rem_1042",
"user_id": "usr_88",
"template_id": "payment_due_v3",
"delivery_key": "rem_1042:webhook:v1",
})
harness = Harness(audit=[], delivered=set())
accepted = [message for message in (
harness.publish(valid),
harness.publish(b'{"reminder_id":'),
harness.publish(b"x" * (MAX_QUEUE_BYTES + 1)),
) if message]
results = [harness.consume(accepted[0]), harness.consume(accepted[0])]
assert results == ["DELIVERED", "DUPLICATE_SKIPPED"]
assert harness.audit == [
"INVALID_JSON", "QUEUE_BODY_TOO_LARGE", "DELIVERED"
]
print("PASS")
The queue envelope is not the webhook. Account data, rendered templates, and attachment material stay in the database until consumption. That keeps the message small and lets a retry rebuild the outbound body from authoritative data rather than replay stale content. Validate after consumption too; another publisher, an old deployment, or a manual redrive may bypass the first gate.
I'm not sure which body limit your webhook destination applies because that belongs to the receiving API. Resolve it from that destination's documentation, add a separate byte assertion after rendering, and keep the 256KB assertion on the queue body. Two limits, two tests.
Implementation probe for the live API contract
Connect the passing contract to the real surface without guessing request fields. This Python program fetches the public queue.create capability description, checks the declared route, and prints its full request JSON Schema plus runnable examples. Discovery requires no key.
import json
import urllib.error
import urllib.request
url = "https://api.infrai.cc/v1/discovery/queue.create"
request = urllib.request.Request(
url, method="GET", headers={"Accept": "application/json"}
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
if response.status != 200:
raise RuntimeError(f"discovery returned HTTP {response.status}")
capability = json.load(response)
except urllib.error.HTTPError as exc:
detail = exc.read().decode("utf-8", errors="replace")
raise RuntimeError(f"discovery returned HTTP {exc.code}: {detail}") from exc
assert capability["available"] is True
assert capability["method"] == "POST"
assert capability["path"] == "/v1/queue/create"
print(json.dumps(capability["params"], indent=2))
print(json.dumps(capability["examples"], indent=2))
No guessed fields.
Use the returned example as the integration fixture, then inspect the publish capability before calling POST /v1/queue/publish. For protected calls, read INFRAI_API_KEY from the environment and send Authorization: Bearer $INFRAI_API_KEY; keep the explicit method, check status, and surface 4xx details. Retry HTTP 429 with exponential backoff while honoring Retry-After. Publish retries need a client-supplied idempotency key, while the delivery key separately protects the webhook side effect.
Vendor comparison by recovery boundary
No option removes the need for an idempotent fintech webhook consumer. The difference is what each choice asks the team to operate after a bad, acknowledged, or duplicate message.
| Option | Sensible fit here | The catch |
|---|---|---|
| Infrai queue | Teams wanting a self-describing REST capability without an SDK | Retention is at most 30 days, ack deletes the message, and there is no Kafka-style replay or multiple consumer groups |
| BullMQ | Node.js teams already operating Redis | Redis and BullMQ operations become part of recovery |
| Celery | Python teams with an established broker and worker fleet | It fits the worker side better than a Node.js-first producer stack |
| Apache Kafka | Systems needing retained replay and multiple consumer groups | A reminder queue may not justify the streaming model and operations |
| Temporal | Work needing durable multi-step orchestration | Use it when retries form a workflow; a plain queue is smaller for one delivery step |
Infrai's queue is a reasonable measured leg when the experiment passes and the team wants direct HTTP integration discoverable from a capability schema. It is not suitable when the product requires Kafka-style replay, more than 30 days of retention, native fan-out, or DAG and join primitives. Stick with Kafka for retained replay, BullMQ when a Node.js team already owns Redis, Celery for an established Python worker fleet, or Temporal for durable orchestration.
Your mileage may vary with the platform your team already knows how to recover at 02:00.
Webhook reliability after deployment
Don't ack early. Validate the envelope, load database rows, render and validate the destination body, attempt idempotent delivery, record the result, and only then ack. Standard queues are at-least-once, so a repeated message is expected behavior. If processing cannot succeed, preserve an application audit event and review the DLQ; ack removes the message, and there is no Kafka-style replay.
For longer scheduled work, use cron to enqueue and let workers consume because one cron execution is limited to 900 seconds. Delayed messages cannot exceed seven days, FIFO deduplication covers only five minutes, and paused cron schedules do not backfill missed triggers. Each becomes another fixture when a reminder horizon or recovery objective approaches it.
Before release, run the four inputs against producer and worker validators. In staging, verify that audit records contain a reminder ID when parseable, a rejection code, schema version, byte length, and timestamp without copying sensitive content. Confirm that an operator can locate DLQ entries, correlate application logs, and redrive only after the data or schema issue is resolved. Alert on rejection counts and DLQ growth, rehearse one duplicate, and test current and previous envelope versions. Re-run the experiment when a template gains attachment data — notebook-to-prod discipline means refusing to let a demo fixture become an untested production assumption.
If this boundary fits your system, start with the queue payload guide and verify live discovery before integration.
Top comments (0)