When designing enterprise workload scheduling systems, one question repeatedly appears:
How can a scheduler handle millions of tasks while keeping low latency and high reliability?
At first glance, task scheduling looks simple:
- Receive a task request;
- Send it to an execution node;
- Wait for the result;
- Update the status.
However, this model becomes much more complicated when the environment grows to hundreds of thousands or millions of tasks;
During the design of WLOADCTL, we focused on one key problem:
How can we decouple task scheduling from task execution while maintaining real-time state awareness?
This led us to an event-driven asynchronous communication architecture based on Message Queue (MQ), Shared Memory (SM).
The Problem with Traditional Communication Models
Synchronous Request-Response
The simplest communication model is synchronous communication:
Client
|
| Request
↓
Server
|
| Response
↓
Client continues
This model works well for scenarios requiring immediate feedback like authentication, payment confirmation and transaction validation.
However, it introduces a major limitation for workload scheduling: The scheduler must wait for execution results. And for long-running tasks the scheduler thread remains blocked, reducing overall throughput.
The Limitations of Database Polling
A common asynchronous solution is database polling. Scheduler periodically queries "Is the task finished?" by following workflow below
Operation System
|
|
↓
Middleware (Message Queue, Shared Memory)
|
|
Component Layers (NLS, FDC, DSY..)
This approach is simple and reliable.
However, when task volume increases to a really big level, several problems appear.
1. Excessive Database Queries
When the database receives a large number of status queries, while most results remain unchanged.
The database becomes a communication layer instead of a storage layer.
2. Additional Latency
Polling introduces a trade-off:
Short interval: More queries, Lower latency and Long interval: Less database pressure and higher response delay
The scheduler always has to choose between performance and freshness.
Event-Driven Architecture and Workflow
In this asynchronous task distribution model, Process A (Scheduler/Sender) and Process B (Executor/Receiver) communicate through a hybrid mechanism combining Message Queue (MQ) and Shared Memory (SM).
Request Queue: Sending Messages to the Queue：Process A encapsulates task metadata, including the unique task ID, pointers/offsets in shared memorydata length, and other related information, into lightweight messages, then sends them in batches to the Request Queue (Message Queue).
Writing Data into Shared Memory: Process A writes large data blocks involved in the task into Shared Memory. This avoids the performance overhead caused by transferring large messages directly through the queue.
No Waiting Required: After Process A finishes sending the task message, it immediately returns without blocking and continues dispatching the next batch of tasks. The execution progress of tasks does not affect the scheduling throughput of Process A.
Slow Execution Does Not Affect Scheduling: Even if Process B processes tasks slowly, it only affects that specific executor. Process A can continue submitting new tasks into the queue with whole scheduling process remains unaffected.
Executor Reads and Processes Tasks: Process B retrieves task metadata from the Request Queue through a polling loop. Based on the pointer information contained in the metadata, Process B reads the actual data from Shared Memory and performs task processing.
Asynchronous Result Feedback: After Process B completes the task, it writes the result data back to Shared Memory (or a new Shared Memory region). It then sends the result metadata, including the task ID and result pointer, to the Response Queue. Process A asynchronously collects results through an independent polling thread or event loop.
Why Combine Shared Memory (SM) + Message Queue (MQ)?
| Component | Responsibility | Advantage |
|---|---|---|
| Message Queue (MQ) | Transfer lightweight control information (task ID, pointer, status) | Decoupling, reliable message delivery, batch distribution |
| Shared Memory (SM) | Transfer large data blocks (actual task content and result data) | Near-zero-copy, high performance, low latency |
The core advantage of this mechanism is the complete decoupling between scheduling and execution.
Through the layered design of MQ + SM, the system achieves both:
Reliable message delivery through Message Queue
High-performance data transmission through Shared Memory
Process A is only responsible for "sending instructions" and does not need to care about:
Who executes the task
How long the execution takes
Whether the task has been completed
Meanwhile, data transfer is carried out through Shared Memory. With no database involved, the overall process is lighter and more performant.
4. Practical Application: High-Performance End-of-Day Batch Processing for a Mid-Sized Bank
In the end-of-day batch processing scenario of a mid-sized bank, the scheduling scale reaches more than hundreds of thousands of jobs across thousands of nodes.
When traditional scheduling platforms handle batch tasks at this scale, merely calculating whether all hundreds of thousands of jobs meet their execution conditions can take up to 4 hours. This means that before the batch window even officially opens, the scheduling engine itself has already consumed nearly half of the night's time window.
In contrast, WLOADCTL's measured performance at this bank was just <10 minutes for the same level job scheduling.
From 4 hours down to <10 minutes, the ultimate impact is: the batch window is maximally released to actual business execution, rather than being wasted on the scheduling engine's own "housekeeping" calculations. For a mid-sized bank, this means that within a single night's batch window, the system can support more business scenarios, accommodate more complex dependency relationships, and reserve ample computational headroom for future business growth.
Top comments (0)