Build a Durable Scheduled-Task Pipeline with n8n and a Cloudflare D1 Queue

I run a small content-publishing pipeline. I write a batch of articles ahead of time, then let a scheduler publish one on a cadence so my feed stays alive without me touching it every day. The naive version of this is a folder of files and a cron job that greps for "the next one." That falls apart the moment two runs overlap or a run crashes halfway through: you either publish the same article twice or skip one entirely.

What I actually want is a queue. Load items once, have a scheduler pop exactly one, process it, and never see it again. In this tutorial I will build that pattern with three pieces: a Cloudflare D1 table as the queue, a Worker endpoint that pops from it, and an n8n schedule trigger that hits the endpoint on a cron. The technique generalizes to any "do one queued thing on a timer" job. I will use publishing as the running example, but the mechanics are the point.

The design in one paragraph

A queue table holds pending items. A pop operation atomically selects the oldest pending row, hands it back to the caller, and deletes it in the same transaction. Because the delete happens as part of the pop, an item can be handed out at most once. That is the tradeoff to understand up front: this is at-most-once delivery. If your processing step fails after the pop, that item is gone. I will come back to why I accept that and how to change it.

Step 1: Create the D1 queue table

Create a D1 database and a table. The schema is deliberately boring.

CREATE TABLE IF NOT EXISTS publish_queue (
  id          INTEGER PRIMARY KEY AUTOINCREMENT,
  payload     TEXT NOT NULL,
  created_at  INTEGER NOT NULL DEFAULT (unixepoch())
);

payload is a JSON blob describing one unit of work, for me the article slug and body. created_at gives me a stable ordering so I drain oldest-first (FIFO). That is the entire queue.

Provision it with Wrangler:

wrangler d1 create publish-queue
wrangler d1 execute publish-queue --remote --file=./schema.sql

Step 2: Load items once

Loading is just inserts. I do this in a batch whenever I finish writing a set of articles. You can load from a script, from the D1 console, or from another Worker route. A simple bulk load from the CLI:

wrangler d1 execute publish-queue --remote \
  --command "INSERT INTO publish_queue (payload) VALUES ('{\"slug\":\"n8n-d1-queue\",\"title\":\"...\"}');"

The important discipline: load is a separate action from drain. You fill the queue in bursts, and the scheduler empties it slowly. The two never race because they touch the table at different times and, as we will see, the pop is atomic anyway.

Step 3: The Worker that pops

Here is the core. The Worker exposes one endpoint. It reads the oldest row, deletes it, and returns it. The trick to making this safe is the RETURNING clause: D1 (SQLite) can delete and return the deleted row in a single statement, so there is no window between "I read it" and "I deleted it" where a second caller could grab the same row.

export default {
  async fetch(request, env) {
    // Only allow POST /pop
    const url = new URL(request.url);
    if (request.method !== "POST" || url.pathname !== "/pop") {
      return new Response("Not found", { status: 404 });
    }

    // Simple shared-secret auth so only your scheduler can pop.
    const auth = request.headers.get("authorization");
    if (auth !== `Bearer ${env.QUEUE_TOKEN}`) {
      return new Response("Unauthorized", { status: 401 });
    }

    // Atomic pop: delete the oldest row and return it in one statement.
    const row = await env.DB.prepare(
      `DELETE FROM publish_queue
       WHERE id = (
         SELECT id FROM publish_queue
         ORDER BY created_at ASC, id ASC
         LIMIT 1
       )
       RETURNING id, payload`
    ).first();

    if (!row) {
      // Queue empty. 204 tells the scheduler there was nothing to do.
      return new Response(null, { status: 204 });
    }

    return Response.json({
      id: row.id,
      payload: JSON.parse(row.payload),
    });
  },
};

A few things worth calling out. The DELETE ... WHERE id = (SELECT ... LIMIT 1) RETURNING form is what gives us the single-statement pop. Ordering by created_at ASC, id ASC makes it FIFO with a deterministic tiebreak. And the empty queue returns 204 No Content rather than an error, which keeps the scheduler quiet when there is nothing to publish.

The QUEUE_TOKEN is a shared secret. Never hardcode it. Set it as a Worker secret and it stays out of your code and out of your logs:

wrangler secret put QUEUE_TOKEN

Bind the database in wrangler.jsonc:

{
  "name": "publish-queue-worker",
  "main": "src/index.js",
  "compatibility_date": "2025-01-01",
  "d1_databases": [
    {
      "binding": "DB",
      "database_name": "publish-queue",
      "database_id": "<your-database-id>"
    }
  ]
}

Deploy:

wrangler deploy

Step 4: The n8n schedule trigger

Now the scheduler. In n8n I use a Schedule Trigger node wired to an HTTP Request node. The Schedule Trigger fires on a cron, and the HTTP Request node calls /pop.

Configure the Schedule Trigger with a cron expression. For once a day at 09:00:

0 9 * * *

Then an HTTP Request node:

• Method: POST
• URL: https://publish-queue-worker.<your-subdomain>.workers.dev/pop
• Authentication: Header Auth, header Authorization with value Bearer <token>. Store the token in an n8n credential, not in the node body, so it does not sit in the workflow JSON.
• Options: set "Never Error" or handle the 204 explicitly so an empty queue does not mark the run as failed.

After the HTTP Request node, add an IF node that checks whether a payload came back. On 204 there is no body, so you branch to a no-op and end the run. When there is a payload, you continue to whatever does the real work: for me, a node that takes payload.slug and payload.body and posts to my publishing platform's API.

That is the whole pipeline. Every morning n8n wakes up, pops one article, publishes it, and goes back to sleep. The queue shrinks by one. When it is empty, the run is a clean no-op until I load the next batch.

The one honest caveat

This design is at-most-once, and you should choose it on purpose. Because the pop deletes the row before your processing node runs, a failure after the pop, say your publishing API is down, loses that item. For my content pipeline that is genuinely fine: the worst case is one article does not go out today, I notice it is missing, and I re-insert it. Low stakes, simple system.

If your items are not disposable, do not reach for this as-is. The standard fix is to make pop a two-phase operation: instead of deleting, add a status column and a leased_until timestamp, flip the row to processing on pop, and only delete after the caller confirms success (an explicit /ack call). Unacknowledged rows whose lease expires get returned to pending by a sweep. That buys you at-least-once delivery at the cost of a lot more moving parts and the need to make your processing idempotent. Know which one you need before you build it. For a solo publishing feed, delete-on-pop is the right amount of engineering.

Wrapping up

The pattern is small on purpose: a table, one atomic SQL statement, and a cron that pokes an endpoint. No queue service to run, no locks to reason about, no state machine unless you decide you need one. D1 gives you the atomic pop for free through DELETE ... RETURNING, and n8n gives you the timer without writing a scheduler.

I use this to keep my writing shipping on a cadence while I work on other things. If you want to see the kind of projects it feeds, my work is at github.com/royalpinto007. Steal the pattern, wire it to whatever you need drained on a timer, and let the cron do the boring part.