The Problem
When it comes to dealing with the problem of background task processing in a distributed .NET system, you are usually presented with two options.
Message brokers (such as Kafka, RabbitMQ, SQS, and others) scale well and are capable of handling high throughput, but they were not designed to provide the kind of answers that a job system needs to give. For example, what actually happened to this particular job? Did it retry and, if so, how many times? Is it still running or did it fail silently three hours ago? These brokers offer delivery guarantees, not an audit trail. If you want to reconstruct what happened, you generally have to piece together the logs from each of the consumers that processed the message. Retries occur at the transport level (that is, through redelivery), not at the job level: the requirement to 'retry this job up to three times with backoff and then mark it as failed so that I can query the reason' has to be built from scratch each time, on top of the broker. Similarly, recurring or scheduled work, for instance, 'run this every day at 2am', is not something that brokers do by default; that too requires an additional layer to be added on.
Traditional .NET job schedulers such as Hangfire and Quartz.NET solve exactly what brokers lack: retries, recurring schedules, and a clear audit trail for every job that runs. But they scale by having each worker instance directly poll the same database table for available work. Beyond a certain point, adding more workers doesn't increase throughput because it creates more processes competing for the same rows. The typical solution "just run more instances" doesn't actually overcome this problem: if you direct all of them to the same table, you'll encounter the same limit with even more competitors; if you shard the database, however, the audit trail that you depended on becomes scattered among the shards and is no longer centralised anywhere.
Brainstorming the Solution
With that issue, I started to think of a way to figure out this dilemma and have a solution that we can audit a job and, at the same time, have high performance. The first thing that came to mind was: why not separate the auditing database from the execution pool? Why do not we divide the jobs into partitions and have a particular worker that owns this partition? Then I started to think about it. It fixes the lock contention problem, but we still need to store the job in a central table. Also, if I schedule a job for a few days, will I provision it into a partition and wait that long? At this point, I was jogging every day, thinking about how to architect this.
The Bucket System Solution
I started thinking about storing the job in a partition for execution, while asynchronously recording it in the audit database.
We have an ephemeral "Agent" layer for execution and transport, and a Master DB for audit history and coordination. I call it an "Agent DB" loosely; it doesn't have to be a database at all. It can be a message broker like NATS, and a single cluster can run multiple Agent connections side by side. The Master DB serves as the long-term store; the Agent layer only ever holds jobs that are actually about to run.
We can also use the bucket system to write to the Master DB asynchronously and increase throughput, since the Agent database only ever contains transient, short-lived jobs. But then: what happens if I schedule a job for next year? It can't sit in a bucket for twelve months.
The answer was to always write the job into a bucket first, then asynchronously persist it to the Master DB from there. That bucket write is what keeps throughput high; nothing waits on a database round trip up front. If the job's execution time is far away, it gets evicted from the bucket once it's safely in the Master DB, and sits there alone until its execution time gets close. Only then does it get moved back into a bucket, where a worker can actually claim and run it. So the bucket isn't a place jobs live long-term; it's the entry point for every job, and the last-mile staging area right before execution. Everything in between belongs to the Master DB.
Yes, there's still some lock contention on the Master DB, but it's contention of a different kind than what we started with. Moving jobs from the Master DB back into buckets as their execution time approaches happens as a large batch operation, not a per-job query fighting for the same rows one at a time. And it only applies to jobs that were scheduled far in advance in the first place; most jobs go straight through a bucket without ever needing this round trip, so the Master DB's write load stays proportional to how much far-future scheduling you actually do, not to your total job volume.
The Architecture
Let's jump to the point and explain the architecture.
Save Process
Every job is written into a bucket immediately, before checking when it's actually due to execute. The TransientThreshold configuration then decides whether the job stays in that bucket for near-term execution or gets moved to wait on the Master DB instead. By default, it's 10 minutes, but you can tweak it to suit your workload.
Assign to Buckets Mechanism
Once a job's execution time gets close, a Coordinator picks it up from the Master DB and assigns it to the bucket matching its priority and worker lane. Each worker owns a set of buckets, one per priority level, so a job dispatched to Worker A's Critical bucket never contends with anything happening on Worker B or C. The assignment itself is exclusive, so no two workers can claim the same job.
If a job fails, it isn't retried in place. It gets sent back to the Master DB, held there, and re-dispatched to a bucket again once it's ready for its next attempt, up to the configured retry limit. Either way, the outcome, succeeded or failed, is recorded back to the Master DB, so the full history stays centralised regardless of which worker actually ran the job.
Try It
JobMaster is open source and currently in alpha. While the integration test suite is robust, I'd like to see more extensive validation under long-running, production-like conditions before considering it truly battle-tested.
If you want to look under the hood, the code is on GitHub: https://github.com/hugoj0s3/jobmaster-net. Docs are here: https://docs.jobmaster.hugoj0s3.dev/. And if you just want to see it running without installing anything, the dashboard sandbox is live: https://sandbox.jobmaster.hugoj0s3.dev/jm-dashboard.
I'd genuinely appreciate feedback, especially from anyone who's hit the same scaling wall with Hangfire or Quartz.NET and worked around it differently.
Final Thoughts
JobMaster is still a work in progress, inspired by real scaling needs and the desire for better visibility into distributed jobs. The bucket system is my current approach, but it's far from the only possible solution.
I welcome feedback, questions, and contributions from anyone solving similar problems or anyone interested in making large-scale job processing more transparent and reliable.
Top comments (0)