Queick: A Simple Job Queue System for Python

Queick is a simple job queue system for Python.

https://github.com/asmsuechan/queick

Feature

• Written in Python only standard libraries
• Job-queue manager without redis
• Working for low-spec machines
• Retry
• Periodic execution
• Retry on network available
• Scheduling
• Thread-base job execution

Main target

Queick can be fully utilized in low-spec computers, such as Raspberry Pi. This system does not aim to be used at the server-side of a normal Web service.

Usage

First, install queick by using pip command.

pip install queick

Second, Second, prepare a job file (jobfunc.py) and an application (test.py).

# jobfunc.py
import time
def function(arg):
    time.sleep(1)
    print(arg)

# test.py
from queick import JobQueue
from jobfunc import function
from time import time

q = JobQueue()
q.enqueue(function, args=("hello",))
q.enqueue_at(time() + 5, function, args=("world",)) # Run after 5 seconds

st = SchedulingTime()
st.every(minutes=1).starting_from(time.time() + 10)
q.cron(st, function, args=(1, 2,)) # Run every 1 minute after 10 seconds

Third, start queick worker.

queick

Finally, run the application.

python test.py

Architecture

As mentioned above, a job is executed on a thread and implemented by concurrent.futures. Also, event notification is used instead of polling to receive jobs.

If a user specifies retry=True option to q.enqueue(), the job will be retried when it fails. The default retry function is Exponential backoff.

Application

Queick is integrated into an entrance recording system made of Raspberry Pi and NFC reader at my laboratory.

This system records the students' entering/leaving history on Slack when they enter and leave by touching their student id.

However, this system has a problem. The Raspberry Pi often disconnects the Internet for some reason. Thus, a job queue system was needed to be integrated to retry the failed post.

I developed a job queue system from scratch because I did not want to install and operate Redis server on the Raspberry Pi

Conclusion

The core of Queick was made in 2 days, yes, my developing speed was quick, therefore, queick. I totally enjoyed developing it.

For further information, check repository asmsuechan/queick.