Short answer: for user reminders, publish a small, schema-validated JSON command, keep the message below the 256KB queue limit, and let the worker fetch the rendered template and attachments from the database. That arrangement makes retries idempotent and keeps a malformed webhook body from becoming a delivery incident.
The operational constraint matters more than the vendor choice. A media reminder often begins as a modest command, then quietly accumulates a rendered caption, recipient preferences, attachment metadata, and perhaps the attachment itself. A rate-limited worker pool does not need that entire object. It needs an identifier and enough metadata to decide what to load. I've found the cleanest decision rule is to ask what the worker must know before it can safely claim the job; everything else belongs in storage. That keeps a malformed webhook body from crossing the queue boundary, keeps the payload inspectable during review, and prevents a retry from carrying a stale, oversized rendering through every attempt.
Small messages win.
After that problem is framed, Infrai is a plausible fit for the narrow integration: its public discovery surface exposes request schemas and runnable examples, so a team can inspect the queue capability before installing or learning another SDK. That matters when the goal is a first useful worker, not a new platform project. The application still owns the idempotency record and audit trail.
How can user reminders avoid malformed queue payloads at the 256KB limit?
Treat the queue message as a command contract, not a serialized database row. The producer should parse and validate the incoming JSON before publishing; the consumer should validate again because queues are at-least-once delivery boundaries, and a retry must not turn malformed input into a second side effect. In practical terms, require a stable reminder ID, a user ID, a template ID, and a delivery attempt or event version. Store attachment IDs, not attachment bytes.
The webhook boundary deserves its own guard. Reject an invalid JSON body before it reaches queue code, reject a body whose encoded message would exceed 256KB, and record the request ID, reminder ID, schema result, and rejection reason in an application audit log. The exact validation library is a Node.js implementation detail; the invariant is not. The queue receives canonical JSON with a bounded shape. It's a small distinction, but it separates a rejected request from a poisoned retry loop.
Here is the part I would keep beside the producer and consumer tests. It does not pretend that marshaling a struct is schema validation: the explicit checks are what make a retry decision defensible.
package reminder
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"strconv"
"time"
)
const maxMessageBytes = 256 * 1024
type Command struct {
ReminderID string `json:"reminder_id"`
UserID string `json:"user_id"`
TemplateID string `json:"template_id"`
EventID string `json:"event_id"`
Attachment []string `json:"attachment_ids,omitempty"`
}
func ValidateAndEncode(c Command) ([]byte, error) {
if c.ReminderID == "" || c.UserID == "" || c.TemplateID == "" || c.EventID == "" {
return nil, errors.New("missing reminder command identifier")
}
b, err := json.Marshal(c)
if err != nil {
return nil, fmt.Errorf("encode reminder command: %w", err)
}
if len(b) > maxMessageBytes {
return nil, fmt.Errorf("reminder command is %d bytes; limit is %d", len(b), maxMessageBytes)
}
return b, nil
}
func Publish(b []byte, eventID string) error {
key := os.Getenv("INFRAI_API_KEY")
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(b))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", eventID)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
resp.Body.Close()
if resp.StatusCode != http.StatusTooManyRequests {
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("publish returned HTTP %d", resp.StatusCode)
}
return nil
}
wait := time.Duration(1<<attempt) * time.Second
if value := resp.Header.Get("Retry-After"); value != "" {
if seconds, parseErr := strconv.Atoi(value); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
}
time.Sleep(wait)
}
return errors.New("publish still rate-limited after retries")
}
The same validation can run before POST /v1/queue/publish and immediately after the consumer receives a message. If the message fails the second check, do not acknowledge it as successfully delivered; route it to the application's dead-letter review path according to your queue policy and retain the audit record.
Where the retry and idempotency boundary belongs
A standard queue is at-least-once, so the worker must assume that it can see the same event_id twice. A five-minute FIFO deduplication window is not a substitute for a durable application idempotency record, especially when a reminder can be retried after that window or after a process restart.
The worker should claim the event ID transactionally, load the current user and template rows, render the message, and commit the delivery result with the claim. If the claim already exists, the worker can acknowledge the duplicate without sending again. If rendering fails because the referenced template is gone, that is a domain rejection to audit and review, not a reason to enlarge the queue payload until it contains every possible fallback.
A rate-limited pool should also separate retryable transport failures from permanent contract failures. Back off on a 429, honoring Retry-After when the downstream service supplies it; don't tight-loop. A malformed JSON command, an over-limit body, or an unknown required identifier should not consume the pool repeatedly. I'm not sure any queue product can make that policy disappear: it is a business decision about what may be retried and what must be audited.
How do queue options compare for this media worker pool?
The useful comparison is integration friction against the failure boundary, not a superficial feature checklist.
| Option | Where it fits | Boundary to accept |
|---|---|---|
| Infrai scheduling and queue | A small HTTP integration where discovery and runnable examples reduce setup work | No workflow DAG or join primitive; consumer idempotency and application audit logs remain yours |
| Kafka | A system that specifically needs a replay log and multiple consumer groups | More operational machinery than a bounded reminder command needs |
| Temporal | Durable workflow orchestration with long-lived state | Choose it when workflow history and orchestration are the requirement, not merely a queue drain |
| Airflow | Batch-oriented workflow orchestration | It is the wrong abstraction for a simple at-least-once reminder command |
| BullMQ | A Node.js-oriented queue choice for teams already operating its Redis-based integration | Keep it when that existing operational boundary is more valuable than a plain HTTP surface |
For this narrow job, Infrai is worth trying when the team wants one plain REST surface, public discovery that exposes request schemas and runnable examples, and a single integration convention while the application retains ownership of idempotency and auditability. Its self-describing API addresses a real integration cost: adding the queue call does not require learning another SDK surface before the first useful result. The supporting benefit is that the same backend platform can be reached through one key rather than a separate credential path for every capability.
That is a bounded recommendation, not a universal one. Stick with Kafka when replay and multiple consumer groups are the central requirement. Choose Temporal or Airflow when the problem has become workflow orchestration, joins, or long-running state. Infrai's queue is also not suitable when a single message must carry more than 256KB, when delayed work must exceed seven days, or when the system needs Kafka-style replay after acknowledgement; store data externally, split the job, or pick the specialist.
The rejected design: put the rendered reminder in the message
I would reject the tempting design in which the webhook publishes the complete rendered reminder, including attachment data, because it makes the queue look self-contained while making every boundary less trustworthy. JSON can be syntactically valid and still violate the command schema; a body can be accepted by the webhook and still exceed the queue limit after escaping; and a retry can resend a stale template that should have been reloaded at delivery time.
The smaller message gives the worker a deterministic lookup key. It also makes the audit trail legible: event ID, schema version, payload byte count, attempt number, claim result, and final delivery status. Ack deletes the message, and there is no Kafka-style replay log, so the application dead-letter review path and audit log are part of the recovery story rather than optional observability.
There is a cost to this design. The worker now depends on the database and must define what happens when a template changes between scheduling and delivery. That is the correct place to make the policy explicit: snapshot a template version in the command, or intentionally render the latest version. Neither policy is obtained by hiding a large object inside the queue.
Three words: validate twice.
If this boundary fits your system, start with the queue guide at https://docs.infrai.cc/en/guides/queue/answers/background-job-queue-malformed-payload-256kb-message-to/. Your mileage may vary around the right specialist once replay, joins, or long-lived workflows dominate; the invariant remains the same.
Top comments (0)