How we stop a bulk social scheduler from double-posting, hammering rate limits, or losing posts

Full disclosure: I build SkedCast, a bulk social media scheduler for agencies. This post is about the boring reliability problems underneath a publishing queue, because they are more interesting than the calendar UI. Nothing here is specific to our product; the same ideas apply to any system that pushes user content to rate-limited third-party APIs.

The job sounds simple: at 09:00, publish this post to N accounts. In practice you have to answer four questions.

1. What happens when the same request arrives twice?

Networks retry. Agents retry. Users double-click. If "create post" is not idempotent, a retry becomes a duplicate on someone's brand account, which is the one bug a scheduler cannot afford.

We layer it:

• The public POST /v1/posts endpoint requires an Idempotency-Key header. Records are kept for 24 hours. Repeating a key while the first request is still running returns 409. Reusing a key with a different body returns 422, so a client bug cannot silently overwrite an earlier request.
• The database backs it up with a unique constraint on (agency_id, idempotency_key), so even if the application layer is bypassed, a second row cannot exist.
• Every publish target has its own idempotency key, and we use it as the BullMQ job id. BullMQ ignores add() for a job id that already exists, so enqueueing the same target twice is a no-op instead of a double publish.
• Bulk creation derives each item's key from sha256(requestKey:index), so a retried bulk import re-creates nothing it already created.

One gotcha worth knowing: BullMQ job ids cannot contain colons, so keys need normalizing (we swap them for underscores) before they are used as ids.

2. How do you avoid looking like a bot without random behaviour?

If you publish the same caption to five Instagram accounts at exactly 09:00:00, platforms notice. The obvious fix is random delay, but random delay makes retries and tests non-deterministic: a retried job would land at a different time than the first attempt.

We use a deterministic jitter instead: fnv1a32(accountId + " " + staggerIndex) % (window + 1). The same account and position always get the same offset, it is never negative, and it involves no Math.random. A retry recomputes the exact same schedule.

The scheduler then applies these steps in order:

1. Start from the later of the requested time and now.
2. Add staggerIndex * minSpacing when spreading is on.
3. Push to at least lastPublishedAt + minSpacing for that account (always applied).
4. Add the jitter.

Each platform has its own defaults for daily cap and minimum interval (for example TikTok is far stricter than Bluesky), and they can be overridden per account.

3. What if the platform says "slow down"?

Rate limits are not failures. We give each platform its own queue (publish:<platform>) so a stuck platform cannot starve the others, and a per-account lease so two workers never publish to one account at once.

The important part is that limits are checked twice: at compose time, and again by the worker at publish time. Between scheduling and publishing, the world changes (another post went out, a cap was hit). If the worker finds the daily cap or spacing exceeded, it defers the target to a rate_limited state rather than failing it, and it waits for the platform's Retry-After when there is one (falling back to 15 minutes). A daily-cap defer waits until the next UTC midnight.

Provider errors are classified rather than blindly retried:

• rate limited: park and wait
• expired token: refresh, then retry
• transient: retry (3 attempts, exponential backoff from 30 s)
• bad request or indeterminate: fail immediately, because retrying a rejected payload only repeats the rejection

Exhausted retries land in a dead-letter queue and fire a post.failed webhook, so a person or an integration finds out.

4. How do you know it actually went out?

This is the part where I want to be careful, because "we confirm it's live" is an easy claim to overstate.

Some platforms let us hand over a post with a future time (platform-native scheduling). For those, a periodic sweep polls the platform to check the post still exists and moves it to published or failed. TikTok has its own status-fetch poll. Anything stuck in platform-side processing for 24 hours becomes indeterminate instead of pretending to be fine.

For ordinary immediate publishes we rely on the platform's API response and reconcile before retrying, rather than running a separate "is it visible" check. Knowing exactly where your guarantees stop is more useful than a vague "verified" badge.

The state machine

Explicit states keep all of this debuggable: draft, scheduled, preparing, publishing, native_scheduled, published, failed, rate_limited, canceled. Every failure path ends in a named state plus a webhook event, never a silent drop.

Takeaways

• Make idempotency a database constraint, not just middleware.
• Prefer deterministic jitter to random jitter.
• Re-check limits at execution time, not only at scheduling time.
• Separate "rate limited" from "failed".
• Write down precisely what you verify and what you don't.

If you want to poke at the API or MCP server behind this, the docs are at https://skedcast.com/developers. Happy to answer questions about any of the design choices in the comments.