Most Python projects I have worked on needed background jobs long before they needed a message broker. A welcome email, a PDF to render, a nightly report. Each time the answer was the same: add Redis, add Celery, add a worker service, then monitor all three.
A few months ago I got tired of it and started tinyjobs, a job queue where the queue is a SQLite file. This post is about how that works: what stops two workers from running the same job, what happens when a worker dies in the middle of one, and where SQLite stops being enough.
A job is a row
from tinyjobs import TinyJobs
app = TinyJobs("sqlite:///jobs.db")
@app.task(max_attempts=5)
def send_welcome_email(user_id: int) -> None:
...
send_welcome_email.enqueue(42)
enqueue inserts one row into a table called tinyjobs_jobs: the task name, the arguments as JSON, the status queued, and a run_at time. That row is the whole message. A worker is a process that looks for rows it can run.
The run_at column does more work than it seems. A delayed job is a row with run_at in the future. A retry is the same row with run_at pushed forward by the backoff. A cron schedule inserts a row at each due time. Three features, one column, and one question for the worker: which rows are due?
Claiming: one statement, one winner
The dangerous moment in any queue is the claim. If two workers both see a job as free and both take it, it runs twice. The classic bug is reading in one statement and writing in the next, because another worker can slip in between.
tinyjobs finds and takes the jobs in a single statement:
UPDATE tinyjobs_jobs
SET status = 'running',
worker_id = ?,
attempts = attempts + 1,
claims = claims + 1,
lease_expires_at = ?
WHERE id IN (
SELECT id FROM tinyjobs_jobs
WHERE status = 'queued' AND run_at <= ? AND queue IN (?)
ORDER BY priority DESC, run_at ASC
LIMIT ?
)
RETURNING *;
It runs inside a BEGIN IMMEDIATE transaction, which takes SQLite's write lock up front. SQLite allows one writer at a time, and that is usually listed as its weakness. Here it is the feature: the lock puts the claims in a line. A second worker waits its turn (up to a busy timeout, five seconds by default), and when it gets the lock its subquery no longer sees the rows the first one took.
The file is in WAL mode, so readers never wait for the writer. Your application, a dashboard or a sqlite3 shell can read the queue while workers claim from it.
When a worker dies
The claim also writes lease_expires_at, sixty seconds ahead. The lease is a promise with an expiry date: "this job is mine until then". While the job runs, the worker renews the lease every twenty seconds.
A dead worker stops renewing. Every worker also runs a small loop, the reaper, every thirty seconds. It looks for rows that are running with a lease in the past, records that attempt as lost, and sets the job back to queued. There is no coordinator and no leader: any worker that is alive does it, and the write lock makes sure only one of them wins each row.
A graceful stop is kinder. On SIGTERM the worker stops claiming, gives running jobs time to finish, and hands the ones it can interrupt back to the queue before it exits, so they don't wait for a lease.
Either way, a job that was interrupted starts again from the beginning. The function's progress lived in the memory of a process that is gone, and the row only says which function to call and with which arguments. So the guarantee is at-least-once: a job is never lost, but it can run twice, and tasks should be safe to repeat.
The answer that arrives late
There is a subtler case. Worker A claims a job and then freezes for two minutes: a paused VM, a laptop lid, a very long garbage collection. Its lease expires, the reaper gives the job to worker B, and then A wakes up and tries to record "success".
Every write of an outcome checks who holds the job:
UPDATE tinyjobs_jobs
SET status = 'success'
WHERE id = ? AND status = 'running' AND worker_id = ? AND claims = ?;
claims is a counter that goes up on every claim, so an old attempt can't be mistaken for the current one, even on the same worker. A's write matches zero rows and changes nothing. The job belongs to B now.
Does it hold?
I didn't want to trust the reasoning alone, so I ran it for ten hours on a small shared VPS: one SQLite file, three worker processes, 25 jobs per second, a graceful restart every fifteen minutes and a SIGKILL every twenty.
897,300 jobs, 24.9/s sustained, zero enqueue failures
68 worker restarts, 29 of them by SIGKILL
nothing lost
Worker memory went from 37.1 MB in the first hour to 37.3 MB in the last, and the p95 from enqueue to finish stayed around 400 ms. For raw throughput, four worker processes drain about 5,200 no-op jobs per second from one file on a local SSD. The limit is fsync, not Python.
You don't have to take my word for it: tinyjobs verify reruns the experiment on your own machine, kills included. The full numbers are on the Measured page.
Two things a broker can't give you
The queue is a database you can read.
sqlite3 jobs.db "SELECT task, status, attempts FROM tinyjobs_jobs WHERE status = 'failed';"
A job can commit with your own data. If the queue lives in the file your application already uses, a job can be enqueued in the same transaction as your rows:
with connection:
connection.execute("INSERT INTO orders (id) VALUES (?)", (42,))
send_receipt.with_options(tx=connection).enqueue(42)
If the transaction rolls back, the job never existed. If it commits, the job is there. No order without a receipt, and no receipt for an order that failed.
Where SQLite stops
It would be dishonest to stop here, so:
- One machine. The file has to be on a local disk. SQLite's locking over NFS or SMB is unreliable, so workers on several servers can't share it.
- One writer. Thousands of jobs per second is the ceiling, not the starting point.
- Workers poll. SQLite can't wake a process up, so an idle worker checks the file, backing off to twice a second.
When you hit one of these, the same tasks run on Postgres or Redis by changing the URL:
app = TinyJobs("postgresql://app@localhost/shop")
On Postgres the claim uses FOR UPDATE SKIP LOCKED and workers are woken with LISTEN/NOTIFY instead of polling. The tinyjobs move command carries the jobs still to run from one store to the other.
That's the whole trick
A table, one atomic statement, a lease, and a check before every write. Most of the work went into the cases around them: shutdowns, signals, clocks and crashes at the worst possible moment.
tinyjobs is on GitHub and PyPI, with no dependencies on SQLite. It can run inside a FastAPI process and works as a django.tasks backend.
If you have run a queue on SQLite, or decided not to, I'd like to hear what you ran into.
Top comments (4)
Really interesting approach😍 I like how you keep things simple while still handling concurrency and reliability Great reminder that sometimes SQLite is all you need that's why i worked with it in django also i gave star nice ⭐
Thank you, and thanks for the star! ⭐ Since you already run Django on SQLite: tinyjobs works as a django.tasks backend, so the queue can live in the same file as your app and a job can commit in the same transaction as your models. If you try it, I'd like to hear how it goes.
Thanks for the suggestion I actually used to work with Django, but deployment has become quite challenging for me due to the restrictions we face in Iran I also never really got into the WordPress trend 😅 These days, I'm exploring Node.js and Next.js instead Still tinyjobs sounds interesting and I might give it a try sometime
That sounds hard, I hope deployment gets easier for you. Good luck with Node.js and Next.js, and thanks again for reading!