A scheduler can get a lot of things slightly wrong and survive. Publishing late is not one of them.
The first version: a cron job
The simplest design is a job that wakes up every minute, finds posts that are due, and publishes them. It works, and Clienzo started there.
Its weaknesses show up with scale. Every minute it asks "is anything due?", mostly to hear "no". When many posts are due at once — 9:00 on a Monday is popular — one run has to work through all of them, and the last one goes out late.
One task per post
Today, when you schedule a post, Clienzo creates a queued task for that one post, set to fire at that exact time. The queue (Google Cloud Tasks) wakes the server only when there is something to do, and posts due at the same minute run side by side instead of in a line.
Publishing runs on its own service, separate from the API the app talks to, so a burst of publishing never slows down the app.
Why the cron job is still there
I kept the cron job, now as a safety net. It covers what a queue cannot:
- posts scheduled further ahead than the queue accepts,
- posts written straight into the database by scripts,
- a task that failed to be created,
- a publish that crashed halfway and needs to be picked up again.
Two mechanisms for one job sounds wasteful. In practice it means a missed task costs minutes, not a missed post.
Never post twice
The dangerous failure for a scheduler is not missing a post. It is posting it twice. So publishing is written to be safe to repeat: before sending a post to a network, the job checks whether that network already has a successful result, and skips it if so.
Stream, don't buffer
Videos were the other problem. The first publishers loaded each file into memory before uploading it, which is fine for a photo and not fine for a long video on a small server.
The publishers now stream media from storage to the network as it is read. In a test with a 600 MB file, memory stayed flat at roughly 80 MB above normal. That test also caught a subtle bug — a download being cancelled because nothing held a reference to it any more — which is exactly why it was worth writing.
Next: what happens when a post goes out on five networks and fails on the sixth.
Originally published on jonasnguyen.com, where I write about building Clienzo and the next startups.
Top comments (0)