DEV Community

Cover image for Queick: A Simple Job Queue System for Python
asmsuechan
asmsuechan

Posted on • Updated on

Queick: A Simple Job Queue System for Python

Queick is a simple job queue system for Python.

https://github.com/asmsuechan/queick

capture

Feature

  • Written in Python only standard libraries
  • Job-queue manager without redis
  • Working for low-spec machines
  • Retry
  • Periodic execution
  • Retry on network available
  • Scheduling
  • Thread-base job execution

Main target

Queick can be fully utilized in low-spec computers, such as Raspberry Pi. This system does not aim to be used at the server-side of a normal Web service.

Usage

First, install queick by using pip command.

pip install queick
Enter fullscreen mode Exit fullscreen mode

Second, Second, prepare a job file (jobfunc.py) and an application (test.py).

# jobfunc.py
import time
def function(arg):
    time.sleep(1)
    print(arg)

# test.py
from queick import JobQueue
from jobfunc import function
from time import time

q = JobQueue()
q.enqueue(function, args=("hello",))
q.enqueue_at(time() + 5, function, args=("world",)) # Run after 5 seconds

st = SchedulingTime()
st.every(minutes=1).starting_from(time.time() + 10)
q.cron(st, function, args=(1, 2,)) # Run every 1 minute after 10 seconds
Enter fullscreen mode Exit fullscreen mode

Third, start queick worker.

queick
Enter fullscreen mode Exit fullscreen mode

Finally, run the application.

python test.py
Enter fullscreen mode Exit fullscreen mode

Architecture

architecture

As mentioned above, a job is executed on a thread and implemented by concurrent.futures. Also, event notification is used instead of polling to receive jobs.

If a user specifies retry=True option to q.enqueue(), the job will be retried when it fails. The default retry function is Exponential backoff.

Application

Queick is integrated into an entrance recording system made of Raspberry Pi and NFC reader at my laboratory.

PXL_20201105_063113362

This system records the students' entering/leaving history on Slack when they enter and leave by touching their student id.

0.ezk306cosec

However, this system has a problem. The Raspberry Pi often disconnects the Internet for some reason. Thus, a job queue system was needed to be integrated to retry the failed post.

I developed a job queue system from scratch because I did not want to install and operate Redis server on the Raspberry Pi

Conclusion

The core of Queick was made in 2 days, yes, my developing speed was quick, therefore, queick. I totally enjoyed developing it.

For further information, check repository asmsuechan/queick.

Top comments (0)