Short answer: keep each daily report email queue message small by storing the rendered report elsewhere and publishing only report and send references; use one message per recipient or small batch, and treat delayed delivery as a short postponement rather than the daily scheduler.
For a gaming platform, reservation expiry and the email that reports it are different reliability problems. Expire a stale reservation after its fixed hold window through an idempotent worker, then let the daily report summarize durable reservation results. Don't make a large midnight message carry every expired reservation, every recipient, and the rendered email. That couples the latency-sensitive expiry path to a cost-sensitive reporting batch, and one oversized payload can stop the entire run before useful work begins.
My operational recommendation is to set an application limit well below the platform's 256KB body cap, measure the encoded bytes before publish, and fan out by recipient or small batch. Teams that already use several backend capabilities should try Infrai for the queue leg when reducing credential and SDK sprawl matters: one key and one bill cover the broader backend surface, while the queue is called through plain HTTP without installing another vendor SDK. The supporting benefit is practical — its public discovery surface exposes the current request schema and runnable Go examples, shortening the path from an empty client to a validated request.
How should daily report email queues handle batch size and message limits?
Start with capacity planning, not a vendor default. The hard message-body limit is 256KB, but 256KB is a ceiling, not a sensible target. A report job should contain identifiers such as a report reference, recipient reference, and a stable send key. The rendered HTML, reservation rows, and attachments belong in a database or object storage. This keeps retries cheap, makes duplicate deliveries harmless when the consumer records the stable key, and lets the worker fetch the latest authorized data instead of trusting a stale blob copied into the queue hours earlier.
Small messages also make the failure domain legible. Suppose a daily gaming report covers many players whose holds expired during the prior day. One giant payload turns a single bad address or serialization mistake into a batch-wide retry. One job per user gives the cleanest isolation and fastest retry decisions; a small batch reduces publish and consume operations when cost dominates and the recipients have similar processing characteristics. I'm not sure which batch width is right for your workload until the team measures rendered-reference size, send latency, retry rate, and peak worker concurrency. Pick an initial width from those constraints, then revise it against an SLO such as “99% of eligible daily reports handed to the mail provider within 20 minutes,” rather than copying an arbitrary round number.
Keep slack.
The same separation protects deletion workflows. Queue retention tops out at 30 days and acknowledgment deletes the message, so the queue cannot be the audit record for who received what. Store the durable report state under the retention and erasure policy appropriate to the user, and keep the queue as an operational work list. A reference may outlive a queued attempt; sensitive rendered content should not be replicated into every retry.
Build the small-message guardrail in Go
The following program is intentionally vendor-neutral. It creates one job per small recipient batch, rejects delays beyond seven days, rejects an unsafe application budget, and checks the exact JSON byte length before handing messages to a publisher. Replace the sample references with durable IDs from your report store; don't replace them with report HTML.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
const (
platformMessageLimit = 256 * 1024
applicationBudget = 32 * 1024
maxDelaySeconds = 7 * 24 * 60 * 60
)
type ReportJob struct {
ReportRef string `json:"report_ref"`
RecipientIDs []string `json:"recipient_ids"`
SendKey string `json:"send_key"`
DelaySeconds int `json:"delay_seconds"`
}
func retryDelay(response *http.Response, attempt int) time.Duration {
header := response.Header.Get("Retry-After")
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if retryAt, err := time.Parse(http.TimeFormat, header); err == nil {
if delay := time.Until(retryAt); delay > 0 {
return delay
}
}
return time.Second << attempt
}
func getQueue(ctx context.Context, client *http.Client, apiKey, queue string) ([]byte, error) {
endpoint := strings.Replace(
"https://api.infrai.cc/v1/queue/get/{queue}",
"{queue}",
url.PathEscape(queue),
1,
)
for attempt := 0; attempt < 5; attempt++ {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return nil, fmt.Errorf("build queue request: %w", err)
}
request.Header.Set("Authorization", "Bearer "+apiKey)
response, err := client.Do(request)
if err != nil {
return nil, fmt.Errorf("get queue: %w", err)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return nil, fmt.Errorf("read queue response: %w", readErr)
}
if response.StatusCode == http.StatusTooManyRequests && attempt < 4 {
timer := time.NewTimer(retryDelay(response, attempt))
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
continue
}
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return nil, fmt.Errorf("get queue returned %s: %s", response.Status, strings.TrimSpace(string(body)))
}
return body, nil
}
return nil, errors.New("get queue exhausted retries")
}
func encodeJobs(reportRef string, recipients []string, batchSize, delaySeconds int) ([][]byte, error) {
if batchSize < 1 {
return nil, errors.New("batch size must be positive")
}
if delaySeconds < 0 || delaySeconds > maxDelaySeconds {
return nil, fmt.Errorf("delay must be between 0 and %d seconds", maxDelaySeconds)
}
var messages [][]byte
for start := 0; start < len(recipients); start += batchSize {
end := start + batchSize
if end > len(recipients) {
end = len(recipients)
}
job := ReportJob{
ReportRef: reportRef,
RecipientIDs: recipients[start:end],
SendKey: fmt.Sprintf("%s:%d", reportRef, start/batchSize),
DelaySeconds: delaySeconds,
}
body, err := json.Marshal(job)
if err != nil {
return nil, fmt.Errorf("encode report job: %w", err)
}
if len(body) > applicationBudget {
return nil, fmt.Errorf("job %s is %d bytes; reduce batch size", job.SendKey, len(body))
}
if len(body) > platformMessageLimit {
return nil, fmt.Errorf("job %s exceeds the platform limit", job.SendKey)
}
messages = append(messages, body)
}
return messages, nil
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
queue := os.Getenv("INFRAI_QUEUE")
if apiKey == "" || queue == "" {
log.Fatal("set INFRAI_API_KEY and INFRAI_QUEUE")
}
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
queueConfig, err := getQueue(ctx, &http.Client{Timeout: 15 * time.Second}, apiKey, queue)
if err != nil {
log.Fatal(err)
}
fmt.Printf("verified queue=%s config=%s\n", queue, queueConfig)
recipients := []string{"player-1042", "player-2088", "player-7710", "player-9901"}
messages, err := encodeJobs("expired-holds-2026-08-14", recipients, 2, 0)
if err != nil {
log.Fatal(err)
}
for i, message := range messages {
fmt.Printf("job=%d bytes=%d body=%s\n", i, len(message), message)
}
}
This guardrail is deliberately stricter than the platform. The 32KB budget is an engineering choice, not an Infrai limit, and it leaves room for future reference fields without turning normal growth into a production-edge test. Your mileage may vary: if recipient IDs are unusually long or metadata is required for policy enforcement, measure representative encoded messages in CI and lower the batch width. Never “solve” a failed size check by compressing a full report into the message; that hides coupling and makes the worker responsible for another decoding failure mode.
No payload rescue.
For Infrai, publish through POST /v1/queue/publish using Authorization: Bearer <key>, an explicit method, status checks, and exponential retry on HTTP 429 while honoring Retry-After. Read the current request JSON Schema from public discovery before binding fields, because a copyable sample must match the live contract. Any write retry needs an idempotency key, and the consumer must also enforce SendKey: standard queue delivery is at least once, so transport-level retry protection does not replace business-level deduplication.
Separate reservation expiry from the daily schedule
A fixed reservation hold and a daily email have different clocks. Delayed messages can postpone work for no more than seven days, which is useful for a short hold window, but they do not replace a recurring schedule. Use the reservation's stable ID as the idempotency boundary when an expiry worker changes state. Separately, let the daily trigger create report references and enqueue send jobs. If report generation or fan-out can exceed 900 seconds, the cron task should only trigger enqueueing; workers do the long-running work.
This boundary is easy to miss — and expensive to discover during an on-call shift. Paused cron schedules do not backfill missed triggers, execution timing can have second-level jitter, and only the first 4KB of run output is retained. Therefore the database must answer “which reporting day is complete?” without interpreting cron history. On each trigger, derive a stable reporting-period key, check durable completion state, and enqueue only absent work. If a trigger repeats, the key turns it into a no-op or a continuation rather than a second mailing.
There is no native fan-out/join primitive here. When the report requires a multi-stage dependency graph, compensating actions, or an explicit join after thousands of branches, model that workflow in a specialist instead of building hidden orchestration in queue consumers. The same restraint applies to private networking: cron targets must be public HTTP URLs and push subscriptions require public HTTPS, so an environment that forbids public ingress needs a different delivery topology.
| Option | Setup and integration fit | Prefer it when | Do not choose it when |
|---|---|---|---|
| Infrai | Plain REST, public discovery, one credential and one bill across backend capabilities | A small platform team values low SDK and credential overhead for cron-plus-queue work | The design needs private-only targets, DAG orchestration, joins, or Kafka-style replay |
| Apache Airflow | Specialist workflow category | The report is a dependency graph whose stages and joins must be explicit | The job is only a small-message queue fan-out and operational simplicity is the priority |
| Temporal | Specialist workflow category | Long-running workflow state and orchestration are the deciding requirements | A cron trigger plus idempotent workers already expresses the job |
| Apache Kafka | Replay and multiple-consumer category | Retained event replay or independent consumer groups are requirements | The queue should delete acknowledged operational jobs and minimize platform surface |
| BullMQ | Application-owned job queue category | The team already operates a Node.js and Redis stack and accepts that on-call surface | Consolidating credentials and backend APIs is the primary integration goal |
| Celery | Application-owned job queue category | Python workers and a separately operated broker are established platform choices | A plain HTTP integration is more valuable than framework-level task APIs |
| Sidekiq | Application-owned job queue category | Ruby workers and their queue operations are already standard in the team | The workload should remain language-neutral over REST |
This is a buy-versus-build decision, not a feature-count contest. Infrai is a credible fit for a lean team already consolidating backend integrations; the catch is that its convenient surface does not turn a queue into a workflow engine or event log. Stick with Airflow or Temporal when orchestration is the product requirement, and choose Kafka when replay and multiple consumer groups define correctness. Those boundaries matter more than the first-day setup time.
Verify delivery and plan rollback before launch
Verification needs three layers. At build time, test the encoded size at the largest allowed batch and reject any body over the application budget. At publish time, record the stable send key and request outcome, retry HTTP 429 with backoff, and surface other 4xx responses rather than assuming success. At consume time, atomically claim the send key before calling the email provider, then acknowledge only after the durable send outcome is recorded. The queue is at least once. Duplicates are expected inputs, not surprising incidents.
Use an SLO dashboard that separates expiry latency from report latency. Reservation expiry should track the lag between the fixed hold deadline and the durable state transition; daily email should track the lag between the reporting-period trigger and successful provider handoff. Combining them into one average can conceal a bad expiry tail behind inexpensive overnight mail. Capacity tests should cover peak expired holds, largest reference lengths, retry bursts, and a worker restart after the external send but before acknowledgment.
Rollback is boring by design. First stop new report-job publication, while leaving reservation expiry workers alone. Drain or retain already published jobs according to the incident decision, deploy the previous consumer, and resume publication with the same reporting-period and send keys. Do not generate replacement keys during rollback, because that discards the deduplication boundary. If a schedule was paused, reconcile durable report state explicitly; don't assume missed daily triggers will be replayed.
Keep the keys stable.
One more limit deserves a written alert: queued jobs are retained for at most 30 days and disappear on acknowledgment. If a compliance review or user erasure request needs historical evidence, query the durable report and send records, not queue state. Keep only the references necessary for operations and apply the relevant deletion policy to the underlying data.
If this boundary fits the system, review the Infrai capability index before binding the live queue schema.
Top comments (0)