I Built a Transactional Background Job Engine for Rust

#rust #backend #systemdesign #showdev

When building backend services, microservices, or CLI tools in Rust, background job execution eventually becomes a hard requirement. Whether you are sending transactional emails, processing AI worker queues, or managing asynchronous database streams, offloading heavy computations out of the main thread is essential.

However, when inspecting the existing ecosystem for job queues across various ecosystems Sidekiq in Ruby, Celery in Python, or BullMQ in Node.js the Rust landscape historically forced engineers into trade-offs:

The Dual-Write Problem: Most distributed job queues mandate an external broker like Redis. If your main database transaction commits but the Redis enqueue step fails due to a network hiccup, your application state becomes permanently out of sync.
Idle CPU Waste: Many SQL-based queue implementations rely on periodic polling loops (SELECT ... LIMIT X every 500ms). Multiply this across hundreds of idle worker pods in Kubernetes, and you bleed CPU cycles and cloud spend.
Engine/Driver Lock-in: Changing your underlying backend from SQLite (on an edge device/desktop app) to PostgreSQL (in production) or Redis (for low latency) usually requires rewriting producer/consumer abstractions.

To address these architectural bottlenecks, I built Azums 🦀⚡: an enterprise-grade, transactional background job queue and streaming framework built natively for the Rust ecosystem.

In this article, I’ll walk through the architectural design of Azums, how it solves the dual-write problem, how it achieves zero idle CPU utilization, and how its performance holds up in micro-benchmarks.

What is Azums?

Azums is a lightweight, zero-cost abstraction for enqueueing, processing, and streaming jobs across multiple storage engines.

Key Highlights

Unified API Across Backends: Write code once using azums, then run it over PostgreSQL, SQLite, Redis, or In-Memory with zero application code changes.
Zero Dual-Write Enqueueing: Enqueue background tasks directly inside your active SQL database transaction block. If the transaction commits, the job queues; if it rolls back, the job disappears.
0.0% Idle CPU Usage: Avoids periodic polling by leveraging native event-driven notifications (LISTEN/NOTIFY in Postgres, PubSub in Redis).
Native Web Framework Extractors: Drop-in JobQueue state extractors for Axum, Actix Web, Poem, and Rocket.
Durable Event Streams: Built-in event log capabilities (consumer groups, offsets, and replayability) inside your database bringing Redis-style streaming into standard relational DBs.
Open Source: Double-licensed under Apache-2.0 / MIT.

Deep Dive: Key Architectural Advantages

1. Eliminating the Dual-Write Problem

Consider a standard e-commerce signup flow. You save a User record to PostgreSQL and want to enqueue a send_welcome_email background job.

// Traditional approach with separate DB and Queue store
let mut tx = db.begin().await?;
user_repo::create(&mut tx, &new_user).await?;
tx.commit().await?; // <--- Point of Failure 1

// If the network drops here, the user exists, but the email is NEVER queued!
redis_queue.enqueue("send_welcome_email", &new_user.id).await?; // <--- Point of Failure 2

azums eliminates this class of bugs through Transactional Enqueueing. Because azums uses your primary database engine, you can pass the active transaction directly into the enqueue call:

use azums::Job;

let mut tx = pool.begin().await?;

// 1. Mutate application data
sqlx::query!("INSERT INTO users (id, email) VALUES ($1, $2)", user.id, user.email)
    .execute(&mut *tx)
    .await?;

// 2. Enqueue job inside the SAME database transaction
client.enqueue_tx(&mut *tx, Job::new("welcome_email", serde_json::json!({ "user_id": user.id }))).await?;

// 3. Atomically commit both user data AND job queue state
tx.commit().await?;

If the transaction rolls back, neither the user record nor the background job is persisted. No dual writes, no distributed locks required.

2. Zero-Cost Idle Execution (Sub-Millisecond Dispatches)

Instead of spinning in a loop issuing queries when the queue is empty, azums leverages database-native notification systems.

┌─────────────────┐       1. Enqueue Job         ┌──────────────────┐
│   Web Application│────────────────────────────>│  PostgreSQL / DB │
└─────────────────┘                              └────────┬─────────┘
        │                                                 │
        │                                                 │ 2. NOTIFY "azums_jobs"
        │                                                 ▼
┌───────▼─────────┐   3. FOR UPDATE SKIP LOCKED  ┌──────────────────┐
│  Azums Worker   │<─────────────────────────────│ LISTEN Receiver  │
└─────────────────┘                              └──────────────────┘

PostgreSQL: Uses LISTEN / NOTIFY. When a job is enqueued, a trigger/notification signals waiting worker threads instantly. Workers claim jobs using high-throughput FOR UPDATE SKIP LOCKED row locking.
Redis: Uses native Redis Pub/Sub combined with atomic Lua scripts.
SQLite: Employs optimized WAL (Write-Ahead Logging) mode and event channels for embedded environments.

This architecture drops idle worker CPU consumption to 0.0% while keeping task dispatch latency under sub-millisecond thresholds.

Code Walkthroughs

Quickstart ("Hello, World!")

Adding azums to your Cargo.toml:

[dependencies]
azums = "0.2"
tokio = { version = "1", features = ["full"] }
serde_json = "1"

Running an in-memory queue worker in under 2 minutes:

use azums::{quickstart, Job};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Connect to storage backend ("postgres://...", "sqlite://...", "redis://...", or "memory")
    let client = quickstart("memory").await?;

    // 2. Enqueue a background job
    client.enqueue(Job::new("greet", serde_json::json!({"name": "Developer"}))).await?;

    // 3. Register a processing handler
    client.register_handler("greet", |job| async move {
        println!("Hello, {}!", job.payload["name"]);
        Ok(())
    }).await;

    // 4. Run worker loop until queue empties
    client.run_until_empty().await?;
    Ok(())
}

Web Framework Integration (Axum Example)

Using azums with web frameworks like Axum, Actix, Poem, or Rocket feels native thanks to custom request extractors:

use axum::{routing::post, Json, Router};
use azums_axum::JobQueue; // Native extractor
use serde_json::{json, Value};

async fn create_user_handler(
    queue: JobQueue,
    Json(payload): Json<Value>,
) -> Result<Json<Value>, String> {
    // Inject jobs directly from route handlers with single-line syntax
    let job_id = queue
        .enqueue_now("default", "send_welcome_email", json!(payload))
        .await
        .map_err(|e| e.to_string())?;

    Ok(Json(json!({ "status": "queued", "job_id": job_id })))
}

#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/users", post(create_user_handler));

    // Serve application...
}

Benchmarks & Performance Metrics

azums runs automated Criterion micro-benchmarks on every commit to ensure zero regressions in execution speed and allocation overhead.

You can inspect the live, interactive Criterion report at the Azums Live Benchmark Dashboard.

Benchmark Target	Operation Context	Throughput / Speed
`enqueue_single_job`	Atomic In-Memory Enqueue	> 100,000 ops/sec
`worker_process_batch_100`	Lease, Attempt, Complete Batch	< 1.5 ms / 100 jobs
`max_throughput_peak`	In-Memory Burst Ingestion	Up to 380,000 jobs/sec

Feature Matrix Comparison

Feature / Metric	Azums 🦀	BullMQ (Node)	Celery (Python)	Sidekiq (Ruby)
Language	Rust 🦀	Node.js	Python	Ruby
Backend Portability	Postgres, SQLite, Redis, Memory	Redis only	Redis, RabbitMQ	Redis only
Idle CPU Usage	0.0%	Low	Medium	Low
Transactional Enqueue	Native (Zero Dual-Write)	No	No	No
Framework Extractors	Axum, Actix, Poem, Rocket	None built-in	Django/Flask integrations	None
Embedded Support	SQLite / Single Binary	Requires Redis process	No	No

Current Project Status & Roadmap

The engine core, multi-backend adapters, and micro-benchmarks are live and stable on Crates.io.

The official documentation is sitting at roughly ~30%, and active development is focused on completing the following roadmap items:

[x] Core PostgreSQL, SQLite, Redis, and In-Memory storage engines
[x] Zero dual-write transactional enqueue API
[x] Native request extractors for Axum, Actix, Poem, and Rocket
[x] Automated Criterion benchmark reporting via GitHub Pages
[ ] Complete the Architecture Book & Low-Level Design (LLD) specs
[ ] Release azums-dashboard (open-source web management console)
[ ] Add OpenTelemetry distributed tracing context propagation

Getting Involved

Whether you are building microservices in Kubernetes, CLI applications, AI agent tools, or single-binary desktop apps, I’d love for you to test out azums and share your feedback!