Why Do Batch Jobs Still Take All Night?

Batch Job Automation with Built-In Job Scheduler

Handling batch jobs are essential in everything from regular data processing and report generation to syncing systems overnight. These jobs often involve predefined sets of steps that need to run at triggered times, and they're expected to do so without manual intervention. Unmeshed makes this easy. In this post, we demonstrate how to automate a complex batch process using the Unmeshed Scheduler.

Use Case: Automating Daily API Aggregation Jobs

Let's say your system needs to collect data from 25 different APIs every night, to process the responses, and later store the results for further reporting. This is a classic complex batch job made up of multiple steps, and it also running in parallel and triggered regularly. Here's how you can build and automate this job in Unmeshed.

Step 1: Define the Process

In Unmeshed, start by creating a new process definition. Then name it something like daily_api_aggregation.

Inside the process, use a Parallel task block to handle multiple API calls at once. We'll start with 5 HTTP steps inside the parallel block, with each calling a different endpoint.

To simulate a more complex job, duplicate the block multiple times. In this case, we duplicated it 5 times, resulting in 25 parallel HTTP steps.

This setup represents a realistic scenario where multiple independent tasks, such as fetching data from different sources, can run simultaneously for efficiency.

Step 2: Test the Performance

Once your workflow is built, you can run a quick test inside Unmeshed.

In our example, the process executed all 25 HTTP steps in under 100 milliseconds.

That's the kind of high-performance orchestration Unmeshed delivers.

Step 3: Schedule the Workflow

Now that your process works as expected, it's time to schedule it.

Unmeshed supports cron expressions. It delivers time-control precision on workflow, whether it's every hour, every Monday at 2 AM, or the first day of every month.

You can create a new schedule in the Unmeshed UI.

Another scheduler feature is the Overlap Policy, which prevents new jobs from starting if the previous one is still running.

If your batch process sometimes takes longer than expected, setting Do Not Allow Overlap ensures that jobs won't run in parallel unintentionally, preventing data conflicts and system strain.

Step 4: Watch It Run

Once saved, Unmeshed takes care of the rest.

In our demo, the job is triggered accurately, ran all 25 steps, and completed in just 55 milliseconds.

You can view all scheduled executions directly in the UI, giving you complete visibility and confidence.

Final Thoughts

Log in to Unmeshed and create your first scheduled process today.