DEV Community: Elias Hernandis

Steady Queue: a Redis-less task backend for Django

Elias Hernandis — Mon, 29 Dec 2025 08:41:00 +0000

This is a cross-post the original article on my website.

A little more than a year ago I resumed using Ruby on Rails professionally after having been out of the ecosystem for several years. The last Rails version I used was Rails 4, and coming back to Rails 8 it was impressive to see how far things had come, especially in terms of the increasing number of tools included by default in the framework. While I was away from Rails, I mostly worked with Django in Python. Although Django is architecturally similar to Rails and also a joy to work with, it provides fewer tools out of the box.

One of the tools that most web apps need is asynchronous task processing. Rails has had a standard interface for defining asynchronous tasks for a few versions already with ActiveJob, but it hadn't shipped with a default backend. Last year, Rosa Gutierrez introduced Solid Queue, which became the default job processing backend in Rails 8. Following the omakase¹ Rails philosophy, it leverages SELECT FOR UPDATE SKIP LOCKED to be able to use a relational database as the concurrency coordination mechanism, eliminating the need to deploy a separate Redis instance.

Django also has excellent options² for async task processing, but, up until very recently, the framework didn't provide a standard interface for how tasks should be defined and the capabilities backends should have. This changed with the adoption of DEP0014 and its corresponding integration into Django 6.0 a few days ago. Jake Howard was the main contributor to both the DEP and django-tasks, the reference implementation. The database backend available in django-tasks uses the same core idea from Solid Queue: leveraging SELECT FOR UPDATE SKIP LOCKED to use a database as the coordinator for concurrency.

I had been wanting to dive deeper into the internals of Solid Queue for a while and having a task interface for Django on track to become part of the framework was the motivation I needed to attempt a port of Solid Queue to Django. Then, Steady Queue was born in mid-August 2025.

A quick tour of Steady Queue

Steady Queue is a database-backed task backend for Django (>= 6.0). Beyond the features already offered by the django-tasks database backend, it also supports:

Recurring tasks (cron-like scheduling)
Concurrency controls (limiting how many instances of a task can run at a time)
Queue management (pausing queues and ordering queues by priority)
Database isolation³, enhanced task argument serialization and a full Django admin interface.

After installing it (it has no dependencies), it can start processing tasks defined with the standard interface available in Django:

from django.tasks import task

@task(priority=10, queue_name='real_time')
def welcome_user(user: User):
    send_email(to=user.email, subject='Welcome to Django!')

Steady Queue extends the decorator-style interface to support limiting how many instances of a task can run at the same time and recurring tasks:

from steady_queue.concurrency import limits_concurrency
from steady_queue.recurring_task import recurring

@limits_concurrency(key='email rate limiting', to=2)
@task()
def send_daily_digest(user: User):
    send_email(to=user.email, subject='Your daily digest')

@recurring(schedule='0 12 * * *', key='send daily digest at noon')
@task()
def daily_digest_at_noon():
    for user in User.objects.all():
       send_daily_digest.enqueue(user)

Configuration is entirely optional, but it supports running multiple workers (and dispatchers and schedulers, depending on your load), prioritizing queues and designating workers to only run tasks from specific queues. Since no Redis is required, all that's needed to start processing tasks is running in your development or production machines

python manage.py steady_queue

We've been using Steady Queue in some production services with light load for a while now, and being able to ditch the Redis deployment was a joy. I'm very happy to take feedback and/or contributions from the community.

Cover photo by Meizhi Lang on Unsplash

Also known as "batteries-included" in the Django world. ↩
Big Huey fan here! ↩
Using a separate database eliminates the possibility of accidentally relying on transactional integrity in tasks. ↩

Defining infinite structures in Haskell

Elias Hernandis — Sun, 20 Oct 2019 21:29:30 +0000

This article is part of a series on Haskell about how easy it is to define infinite structures in really concise ways. Building on the previous article about lazy evaluation, we explore how to define infinite structures using recursion without using a base case. While in other languages this would result in an infinite loop, this does not happen in Haskell due to its powerful lazy evaluation strategy.

First steps: an infinite list of ones

Recall from the previous post that in Haskell, parameters are evaluated only when they are needed. Let us revisit a simple example. Consider the following function definition.

f :: Int -> Int -> Int
f a b = a * 2

Calling f with parameters a = 2 + 3 and b = 3 * 3 would result in the following execution:

  f a b
= f (2 + 3) (3 * 3)
= (2 + 3) * 2
= 5 * 2
= 10

Notice how the value of the parameter b is never computed. More importantly, if the parameter b did not contain a simple arithmetic expression, but rather some thing more complex, Haskell would not waste time computing its value. Moreover, if the parameter b was undefined, i.e. if it could not be computed, the function call f a b would not fail.

Now consider the following definition:

ones :: [Int]
ones = 1 : ones

What do you think would happen if we called ones? Well, ones is defined recursively, but it does not have a base case, so normally we would say that execution does not terminate, or that a maximum depth of recursion is reached and the program stops, or something along those lines. In Haskell, this is still true, calling ones directly will result in the program never stopping.

But, what if we called take 3 ones. Remember that the function take n xs returns the first n elements of the list xs. And indeed,

ghci> take 3 ones
[1,1,1]

This not only doesn't fail, but returns the expected result. Let's look at the definition of take.

take :: Int -> [a] -> [a]
take 0 _ = [] -- base case
take n (x : xs) = x : take (n - 1) xs -- recursive case

And let's look at how Haskell handles the call take 3 ones.

  take 3 ones
= -- n = 3 dictates we go into the recursive case
  -- parameters are passed without being evaluated
take 3 (1 : ones)
= -- in the body of the recursive case,
  -- pattern matching assigns x = 1, xs = ones 
1 : take 2 ones
= -- n = 2 dictates we go into the recursive case
  -- the same pattern matching assignments are made
1 : 1 : take 1 ones
= -- n = 1 dictates we go into the recursive case
1 : 1 : 1 : take 0 ones
= -- n = 0 dictates we go into the base case
1 : 1 : 1 : take 0 ones
= -- this time, the body of the base case does not depend on xs
  -- so it is not evaluated and the function returns
1 : 1 : 1 : []
= -- syntactic sugar
[1,1,1]

Magic! As the expression take 3 ones did not depend on the full, infinite, list of ones, only the relevant part was evaluated and Haskell was able to correctly compute the expected result!

More interesting infinite definitions

Base-case-less recursions need not be that simple. Assume we want to represent all of the natural numbers in Haskell. We could easily do so with

nats :: [Int]
nats = 0 : map (+1) nats

And indeed,

ghci> take 10 nats
[0,1,2,3,4,5,6,7,8,9]

(Exercise: try to expand the call stack for take 2 nats as we did before)

This is a common enough case that Haskell has special syntax¹ for it:

nats = [0..]

-- and we can even start later
[2..]

The sieve of Erathostenes

Erathosthenes was a Greek mathematician who gave an algorithm to find all prime numbers up to a limit. The algorithm is similar to trial division, only it is constructive (i.e. instead of testing wether a number is prime, it finds all prime numbers up to a limit). Lets look at the pseudocode

List all numbers starting with 2 until n, where n is the upper bound in our search for primes.
Take the first element p in the list and remove all multiples of $p$ from the list, i.e. remove 2p, 3p, 4p, ... from the list. Another way to think of this is, remove a number x if mod x p == 0.
Go back to the previous step starting with the first remaining number.

This algorithm may be used without an upper bound if one wishes to find all of the prime numbers. In most languages this would not be possible but it is in Haskell:

-- List all numbers starting with 2, until infinity
primes :: [Int]
primes = sieve [2 .. ]

-- take the first number p in the list
sieve (p : xs) = p : ...

-- remove all multiples of p from the list
sieve (p : xs) = p : sieve [x | x <- xs, x `mod` p /= 0]

-- Done! (the definition above is recursive)

And indeed,

ghci> take 4 primes
[2,3,5,7]

Lazy evaluation is a powerful evaluation strategy that allows us to concisely and expressively write definitions of infinite structures. In future posts, we will explore other consequences of this design decision of the Haskell language and some situations in which it does not prove so useful.

You might have seen the syntax [a..b] before. It is a shorthand for the function enumFromTo defined in the enum class. In fact, [a..] is also a shorthand for the method enumFrom. Integers in Haskell are an instance of the Enum class that defines these methods. ↩

Lazy evaluation in Haskell

Elias Hernandis — Thu, 17 Oct 2019 21:21:19 +0000

Cover image by Andreas Kay licensed under CC BY-NC-SA 2.0.

A version of this article previously appeared in my blog.

This article is part of a series on Haskell about how easy it is to define infinite structures in really concise ways. Lazy evaluation is an evaluation strategy that is the foundation of many features of haskell, from performance to expressiveness. In this first article, we explore different evaluation strategies found in other languages, how they compare to Haskell's and their benefits and drawbacks. No Haskell knowledge is required for this first post :)

Call me by your value

When discussing different families of programming languages, we sometimes use the phrases call-by-value to refer to how parameters are passed to a function when it is called. Consider the following code in a generic language

def f(a, b):
  return a * 2

If we say the language is call-by-value we mean that the variables a and b are evaluated before copies of them are passed to the function f. For instance, if we called f with arguments a = 2 + 3 and b = 3 * 3 the following would happen in a call-by-value language:

  f(a, b)
= f((2 + 3), (3 * 3))
= f(5, (3 * 3))
= f(5, 9)
= 5 * 2
= 10

It takes roughly 5 steps to compute the value f(a, b). Observe that even though the value of f is only dependent on the value of a, the parameter b is still evaluated. We call this evaluation strategy strict, because it does not care about the usefulness of the values, it just computes everything as a very stubborn robot would do. This is the case with some low-level languages
like C.¹

Call me by your name

What if we delayed evaluation until the parameters were actually needed? We call this call-by-name and it is an example of a non-strict evaluation strategy. Let's look at how the previous computation would look like with a call-by-name approach:

  f(a, b)
= f((2 + 3), (3 * 3))
= (2 + 3) * 2
= 5 * 2
= 10

Wow! We saved one step! This is not too much but suppose that a and b weren't simple arithmetic expressions but complex computations instead. Then not computing b would certainly be beneficial. Is this always the case? Unfortunately, it turns out, it isn't. Look at the following definition and compare the two evaluations:

def square(a):
  return a * a

-- Call-by-value (strict) evaluation
  square(a)
= square((2 + 3))
= square(5)
= 5 * 5
= 25

-- Call-by-name (non-strict) evaluation
  square(a)
= square((2 + 3))
= (2 + 3) * (2 + 3)
= 5 * (2 + 3)
= 5 * 5
= 10

Yikes! That's one more step. Now a natural question is, which evaluation strategy makes more sense? Call-by-name seems better since unused values are not computed... Also, why did we calculate (2 + 3) twice? Couldn't we have stored the value for the second calculation?

Call me lazy

Turns out if we add sharing to call-by-name, this strategy will never take more steps to evaluate an expression than call-by-value. We call this call-by-need or lazy evaluation in the context of Haskell. Call-by-need is also a form of non-strict evaluation but has this added benefit of never introducing a performance penalty (well, at least in the number of steps²).

Let's look at how the previous examples would look like if we used call-by-need.

  f(a)
= f((2 + 3))
= (2 + 3) * 2
= 5 * 2
= 10

  square(a)
= square((2 + 3))
= (2 + 3) * (2 + 3)
= 5 * 5
= 10

We can see that in both cases the evaluation takes fewer steps than call-by-name or call-by-value.

Closing words

Call-by-need or lazy evaluation is central to both the performance and expressiveness of the Haskell language. In addition, the Haskell compiler has pureness and a strong type system at its disposal, enabling it to make much more aggressive optimisations than would be possible in other languages. More simply, expressiveness in Haskell does not come with a performance price to pay.

In the following articles we will apply lazy evaluation to create infinite structures.

If you want to read more about evaluation strategies, you can check out this wikipedia article

Although in some languages we may substitute call-by-name with call-by-reference, the idea of strict evaluation still applies — all parameters are evaluated before the function call — only in call-by-reference the address of the value is passed instead of a copy of the value itself, allowing the function to modify it. ↩
It may, however, introduce a memory usage penalty, but these cases are more rare and easier to detect and fix just by rewriting the order of arguments. ↩