DEV Community

Cover image for One Temporal Schedule per customer, or one sweeper?
Jules Robineau
Jules Robineau

Posted on Originally published at jrobineau.com

One Temporal Schedule per customer, or one sweeper?

Your users schedule a task themselves, one that comes back every week, every month or every year. You use Temporal. The first reflex is to create one Temporal Schedule per customer.

I considered it, then dropped it. As soon as the date lives both in your database and in Temporal, you have two sources of truth. And nothing keeps them in agreement.

TL;DR: one Schedule per customer copies the recurrence rule into two systems, with no shared transaction. Keep the recurrence in the database, with its next date. One Schedule starts a sweeper that processes everything due, in batches of 500. If a run is skipped, the next one catches up, because the database knows what is late. In this project, I keep native Schedules for jobs defined in code. The acceptable delay guides that choice.

This article is for Go developers who schedule recurring tasks with Temporal, or who are thinking about it. You do not need to know Temporal in depth: I define every term.

The context: recurrences created while the system runs

In early October, I designed task scheduling for about twenty microservices. The project is in Go, and the chosen engine is Temporal.

There were two kinds of tasks. First, system jobs with a fixed frequency, like a reminder sent every day at 8 am. Then, recurrences chosen by the user.

A recurrence comes back every week, every month or every year. It has a start date and sometimes an end date. It is created and changed through an API, while the system runs.

I ruled out a home-made loop that wakes up every minute to look for work. Temporal does the triggering.

Temporal offers Schedules for that. A Schedule starts a workflow at planned times. It resembles cron, but managed by Temporal. A workflow is a Go function that Temporal records step by step, so it can resume it after a failure.

I built a prototype in Go, with Postgres and Temporal, to compare two options.

The obvious option: one Schedule per customer

The idea is simple. When the API creates a recurrence, it also creates a Schedule just for it. Temporal fires it on every due date, and its UI shows each customer.

The number of Schedules is not a problem in itself. On Temporal's official forum, a 2024 answer says there is no limit.

The Schedules page gives a single load hint. An action usually starts within one second of its planned time. Unless many Schedules fire at the same moment.

So volume was not my problem. The problem was where the date lives.

Two systems, two truths

The recurrence has to live in your database. The API shows it, changes it and knows when it ends. With one Schedule per customer, the same rule also lives in Temporal.

Every change then becomes two writes, in two systems. They share no transaction. If the second one fails, the first one stays.

Take a simple case. A user switches a recurrence from monthly to weekly. The database saves the change. Then the call that updates the Schedule fails on a network cut.

Your database says "every week". Temporal still fires every month. Nothing warns you.

To guard against it, you need a job that compares both sides and fixes the gaps. But Temporal serves the list of Schedules through its search system. The documentation warns that this list is eventually consistent. A Schedule that was just created or deleted may not show up right away.

So your check job compares your database with a list that may lag behind.

A Schedule's calendar cannot say "the last day of the month" either. Temporal's API definition says it plainly. Such relative expressions are not representable, and you must list the days you want.

Yet a recurrence created on the 31st must land on February 28, then return to March 31. I covered that calculation in the AddDate article. With one Schedule per customer, that calculation would stay in your code anyway.

The sweeper: the database is the only truth

The other option keeps everything in the database. Each recurrence carries its next date, next_run_at. A single Schedule starts a sweeper at a fixed interval. The sweeper is a workflow that processes every due date that has arrived.

One Schedule per customer versus one sweeper

The recurrence lives in the database, one sweeper does the rest. At the top, every change writes into two systems. At the bottom, the API writes only to the database, and Temporal just runs on time.

It finds the due rows with this query:

SELECT id, period, anchor_at, end_at, timezone, next_index
FROM recurrences
WHERE status = 'active' AND next_run_at <= now()
ORDER BY next_run_at
LIMIT $1
FOR UPDATE SKIP LOCKED;
Enter fullscreen mode Exit fullscreen mode

Three details matter.

  • next_run_at <= now(), not "equal to today". A due date missed during an outage is still due on the next run.
  • LIMIT $1 caps the batch. The workflow passes the value 500.
  • FOR UPDATE SKIP LOCKED locks the rows it takes. Another sweep running at the same time skips them instead of waiting.

A partial index on next_run_at keeps the query fast. A partial index only covers the rows that match a condition, here status = 'active'.

The API, for its part, only writes to the database. On creation, the first due date is the start date. The API then asks the sweeper to run right away, with Trigger:

// The first run is due now: ask the sweeper to run instead of waiting for its next tick.
// If this fails, the next tick catches up, so it is not an error for the caller.
h := c.ScheduleClient().GetHandle(ctx, "recurrences.sweeper")
if err := h.Trigger(ctx, client.ScheduleTriggerOptions{}); err != nil {
    log.Printf("trigger sweeper: %v", err)
}
Enter fullscreen mode Exit fullscreen mode

Look at the comment. If this call to Temporal fails, nothing is lost. The database already says the due date has arrived, and the next run will process it.

With one Schedule per customer, the same failure would have left both systems in disagreement.

In my design, changing a recurrence means closing the old one and creating a new one. The old one's end date is set to now. Everything happens in the database, and Temporal has nothing to update.

Batches of 500, then ContinueAsNew

Here is the sweeper workflow. Only the activity name differs from the prototype:

const batchSize = 500

func SweepWorkflow(ctx workflow.Context) error {
    ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{StartToCloseTimeout: time.Minute})
    var a *Activities
    for range 100 {
        var processed int
        if err := workflow.ExecuteActivity(ctx, a.ProcessDue, batchSize).Get(ctx, &processed); err != nil {
            return err
        }
        if processed < batchSize {
            return nil // nothing left to do
        }
    }
    // 100 full batches: start again with a fresh history.
    return workflow.NewContinueAsNewError(ctx, workflow.GetInfo(ctx).WorkflowType.Name)
}
Enter fullscreen mode Exit fullscreen mode

An activity is a workflow step that touches the outside world, here Postgres. Temporal retries it when it fails. ProcessDue takes at most 500 due rows, processes them in one transaction and returns their count.

Fewer than 500 rows? Nothing is left, so the workflow stops. Exactly 500? There may be more, so it goes again.

Why stop after 100 batches? Temporal records every step of a workflow in an event history. That history is capped at 51,200 events or 50 MB. Temporal warns you from 10,240 events or 10 MB.

ContinueAsNew restarts the same workflow with an empty history. After 100 batches, so 50,000 rows, the sweeper starts fresh and keeps going.

This exit mostly matters after a long outage, when the backlog piles up.

SKIP is enough, because the database knows what is late

A Schedule has an overlap policy. It says what to do when the time comes while the previous run is still going. Temporal offers six of them:

  • Skip, the default. The new run is not started.
  • BufferOne and BufferAll. The next run, or all the next runs, wait for the current one to finish.
  • CancelOther. The current run is cancelled, then the new one starts.
  • TerminateOther. The current run is killed at once, without waiting.
  • AllowAll. Several runs go on at the same time.

At first, the wish was to cancel the current run and start the new one. That is CancelOther. With a sweeper, I kept Skip.

The reason is simple. The sweep in progress already handles what is due. A skipped run loses nothing, because the next one reads the database again. Cancelling a sweep in the middle of a batch would only throw work away.

Same logic for catching up. A Schedule has a catchup window, one year by default. It covers the actions missed while the Temporal server was down.

The sweeper does not depend on that window. The server may have been down, or the worker, the program that runs your code. A run may also have been skipped. In every case, the query finds the late rows.

In the prototype, a recurrence anchored on January 31 received its 9 missed due dates in a single run.

Careful, the overlap policy only governs the runs started by that Schedule. It is not a lock on your data. The real protection stays in the database. It is the row lock, plus the per-due-date unique index from the AddDate article.

What the sweeper does not do on its own

The sweeper has two limits, worth knowing before you choose.

The first one is delay. A due date may wait until the next run. In my design, the sweeper runs every hour. In the prototype, it runs every 30 seconds, to see the result right away.

Trigger handles the first due date. For the next ones, pick the interval from the delay your product accepts. If each customer needs the exact minute, one Schedule per customer becomes defensible again.

The second limit is isolation. In the prototype, a batch of 500 fits in one transaction. It succeeds or fails as a whole. And by default, Temporal retries an activity with no limit on attempts.

The transaction and unique constraint make SQL retries safe. A permanent SQL error can still block the entire batch. Report it and isolate the failing item. But if each row calls another service, a single outage makes the whole batch fail.

During those retries, the workflow is still running. With Skip, the next runs are therefore skipped. The whole sweep waits on one row.

The prototype has the answer, in another job. That job starts one child workflow per item. A child workflow is a workflow started by another workflow. Its ID comes from the item. For a recurrence, use the recurrence and due date together to distinguish occurrences.

For the parent to continue, it must wait for the child to start, not for its result. Set ParentClosePolicy to ABANDON if the child must outlive the parent. By default, closing the parent terminates the child. Temporal refuses two active executions with the same ID. This does not deduplicate external calls from a retried activity. The called service must recognize the same idempotency key.

I simulated an outage of a service called by that job. Only the failing step was retried, then the child finished with no duplicate.

I would do the same for a sweeper that calls other services. It picks the due rows, then hands each one to its own child.

The rule: who owns the definition?

In the end, I kept both tools, each in its place. The boundary fits in one question: who owns the definition of the task?

  • The code owns it. It is a system job, and a developer sets its frequency before deployment. It gets its own native Schedule.
  • The database owns it. It is a recurrence, and a user creates it while the system runs. It goes through a sweeper, one per recurrence type.

For system jobs, the worker creates each Schedule when it starts. If the Schedule already exists, it updates it with the definition from the code. So the code stays the only truth, like the database for recurrences.

The checklist

The date lives in one place. The other systems only read it.

  • [ ] For each task, ask who owns its definition: the code or the database
  • [ ] Keep the next date in the database, with a partial index on active rows
  • [ ] Find due rows with <= now(), never with "equal to today"
  • [ ] Take rows in batches, with FOR UPDATE SKIP LOCKED
  • [ ] Switch to ContinueAsNew after a fixed number of batches
  • [ ] Call Trigger on creation, and accept that it may fail
  • [ ] Pick the sweeper interval from the delay your product accepts
  • [ ] Give a child workflow to each item that calls another service
  • [ ] Create or update system job Schedules when the worker starts

What to remember

One Schedule per customer looks cleaner, because Temporal takes care of everything. In fact, it copies your recurrence rule into a second system. And then you have to keep both copies in agreement.

The sweeper does the opposite. Temporal knows only one thing: when to run. The database knows everything else. An outage, a skipped run or a failed call then leaves no disagreement to repair.

Are you weighing different ways to schedule tasks in Go, with or without Temporal? Let's talk.


Sources: Temporal, child workflow lifecycle · Temporal, Schedules (overlap, catchup, listing) · Temporal, Schedules in Go · temporalio/api, Schedule definition · Temporal forum, limit on the number of Schedules (2024) · Temporal, Event History limits · Temporal, Continue-As-New · Temporal, retry policies · PostgreSQL, SELECT and SKIP LOCKED · PostgreSQL, partial indexes

Top comments (0)