DEV Community: Yogaraj.S

ScheduledThreadPoolExecutor in Python

Yogaraj.S — Tue, 21 Jun 2022 04:24:36 +0000

Similar to Java APIs provided in ScheduledThreadPoolExecutor, I’ve implemented the same in Python. Read further for a detailed explanation.

Thread Pool

A thread pool is a design pattern in which multiple threads are maintained to perform the tasks submitted to it. Thread pools are used where multiple short-lived tasks need to be executed and also to increase the performance of the application.

Benefits of using a thread pool

Instead of maintaining a thread pool, we can also create a thread for each task as needed. The main benefit of using a thread pool is that we avoid the cost of creating and destroying the threads. From Wikipedia,

Creating too many threads wastes resources and costs time creating the unused threads.

Destroying too many threads requires more time later when creating them again.

Creating threads too slowly might result in poor client performance (long wait times).

Destroying threads too slowly may starve other processes of resources.

ThreadPoolExecutor

A ThreadPoolExecutor is a kind of supervisor program using which the tasks are submitted to the thread pool. Each programming language has some kind of concurrency package where the thread pool executors are provided. For example, concurrency is provided by the java.util.concurrent package, whereas in Python, it’s provided by the concurrent.futures package.

When a task is submitted to a thread pool executor, a thread is assigned to execute the task and return the result. The number of threads in a thread pool is decided by the computational resource available in a computer.

ScheduledThreadPoolExecutor

ScheduledThreadPoolExecutor is an executor service where the submitted tasks are executed with an initial delay and a periodic interval repeatedly.

ScheduledThreadPoolExecutor comes in handy when we need to run a particular task repeatedly at a specific interval. Example: Checking the replication of a component every 10 minutes and triggering a mail if there is slowness or failure.

The implementation of a ScheduledThreadPoolExecutor is simply done by extending the ThreadPoolExecutor and by maintaining a delay queue to provide the tasks based on the interval. Most languages provide the ScheduledThreadPoolExecutor as a part of their concurrency package. Java provides the ScheduledThreadPoolExecutor in its concurrency package by default. Java provides two methods to execute the task periodically.

scheduleAtFixedRate — Creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given period; that is executions will commence after initialDelay then initialDelay+period, then initialDelay + 2 * period, and so on.
scheduleWithFixedDelay — Creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given delay between the termination of one execution and the commencement of the next.

ScheduledThreadPoolExecutor in Python

Python provides two different modules, sched and concurrent.futures and it is in the hands of the user to implement the ScheduledThreadPoolExecutor. Though there are packages that provide scheduling in Python, there is no implementation as close to the one provided by Java. So, I’ve implemented the same with the APIs to closely resemble the ones in Java.

The Python package for ScheduledThreadPoolExecutor is published in PyPI as well. You can install it using pip install scheduled-thread-pool-executor . It provides the three main methods schedule, schedule_at_fixed_rate, schedule_at_fixed_delay similar to the one in Java. I believe this will be useful for many who are in search of such implementations in the future.

syogaraj / scheduled_thread_pool_executor

Scheduled Thread Pool Executor implementation in python

Scheduled Thread Pool Executor

Scheduled Thread Pool Executor implementation in python

Makes use of delayed queue implementation to submit tasks to the thread pool.

Usage

from scheduled_thread_pool_executor import ScheduledThreadPoolExecutor
scheduled_executor = ScheduledThreadPoolExecutor(max_workers=5)
scheduled_executor.schedule(task, 0)  # equals to schedule once, where task is a callable
scheduled_executor.schedule_at_fixed_rate(task, 0, 5)  # schedule immediately and run periodically for every 5 secs
scheduled_executor.schedule_at_fixed_delay(task, 5, 10)  # schedule after 5secs (initial delay) and run periodically for every 10secs

Note

This project has been set up using PyScaffold 4.1.1. For details and usage information on PyScaffold see https://pyscaffold.org/.

View on GitHub

Debugging timezone issue in Java [Linux]

Yogaraj.S — Fri, 18 Mar 2022 08:28:40 +0000

Have you ever noticed that in some cases, Java will not take the time zone which is configured in a server? This has serious impacts if the server handles scheduling tasks and runs them periodically.

Before diving deep down, let’s understand what is a time zone?

Time Zones are geographical world globe division of 15 degree each, starting at Greenwich, England. It’s a uniform standard time for legal, commercial and social purposes.

Issue: Timezone difference between the system and java

While executing the below code, It will print the system time along with the timezone. I’ve added the output of the date command as well.

Even though the system timezone is configured as PST, java prints the timezone as GMT.

Quick fix:

Set user.timezonein the command line as -Duser.timezone=America/Los_Angeles
Set the environmental variable TZ to provide the timezone.

The above solutions will work, but to understand and fix it once for all, read further for the actual fix. (All required source code links are embedded in the method names)

Detailed explanation on why do we get this timezone difference?

In the example java code, we call System.out.println of new Date(). We know that java calls an object’s .toString() method internally to provide the string representation of an object. Let’s take a look at the toString method of the Date class in java.

The timezone on Line:16 is set by calling the date.getZone(). The variable date is initialized by calling the normalize() method.

The timezone in normalize is set by calling theTimeZone.getDefaultRef() in TimeZone.java which then calls setDefaultZone()to get the system timezone.

Now, let's break up and understand what the setDefaultZone() does.

It checks for the zone by checking the user.timezone property (which we didn’t set via the command line).
If the user.timezone property is null, it gets the java.home property and calls the getSystemTimeZoneID() which is a native method.
Any NULL values will set the timezone to default GMT. (So, this is why the timezone is set to GMT!)

Ignoring the unwanted details, let’s focus on what the findJavaTZ_md() does.

Now, we can understand why the quick search solutions mentioned to set values for user.timezone and TZ! In our case, we didn’t set the TZ environmental variable which means TZ = NULL.

It calls getPlatformTimeZoneID() if TZ is NULL. (We’re about to reach a conclusion in the next step! Hang on..)

As the comment says, it will check for /etc/timezone file which contains the system timezone information. I checked in my Ubuntu machine for the file and it's straightforward.

If the file is not present, it checks /etc/localtime to obtain the timezone.

Here’s what happens in the above code.

Get the file status of /etc/localtime using lstat
Check whether it's a symlink and read the symlink to obtain the timezone.

For lstat to work as expected, it's required to have the executable(x) permission on the file (as stated in the manpage of lstat). In my case, the /etc/localtime file doesn’t have the executable permission set. Hence, it returns NULL taking the default timezone of GMT.

Obtaining Timezone via symlink:

If the file has the required permission set, the output will be like the below one. After resolving the symlink, the zone info will be taken using the getZoneNamemethod.

If the /etc/localtimefile is not a symlink, the getPlatformTimeZoneID method will try to open and read the file like /etc/timezone. As the last fallback, it will recursively iterate the /usr/share/zoneinfo directory to find the timezone. When none of them works out, the timezone is set to GMT- the default!

Actual Fix:

For my case, I just had to create a symlink of /etc/localtime from /usr/share/zoneinfo/America/Los_Angeles.

After setting the symlink, running the DateExampleprogram yields the expected result.

Thank you for reading!